MODULI SPACES OF DECOMPOSABLE MORPHISMS OF SHEAVES 
AND QUOTIENTS BY NON-REDUCTIVE GROUPS 



JEAN-MARC DREZET AND GUNTHER TRAUTMANN 



Abstract. We extend the methods of geometrie invariant theory to actions of non- 
reductive groups in the case of homomorphisms between decomposable sheaves whose 
automorphism groups are non-reductive. Given a hnearization of the natural action 
of the group Ant{E) x Aut(F) on Hom(E,F), a homomorphism is called stable if its 
orbit with respect to the unipotent radical is contained in the stable locus with respect 
to the natural reductive subgroup of the automorphism group. We encounter effective 
numerical conditions for a linearization such that the corresponding open set of semi- 
stable homomorphisms admits a good and projective quotient in the sense of geometric 
invariant theory, and that this quotient is in addition a geometric quotient on the set of 
stable homomorphisms. 
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1. Introduction 



Let X be a projective algebraic variety over the field of complex numbers. Given two 
coherent sheaves T ^ on X the algebraic group G = Aut(£^) x Aut(jF) acts naturally on 
the affine space W = Hom(£, !F) by {g,h).w = h o w o g~^. If two morphisms are in the 
same G-orbit then they have isomorphic cokernels and kernels. Therefore it is natural to 
ask for good quotients of such actions in the sense of geometric invariant theory. 



1.1. Morphisms of type {r,s). 

In general S and will be decomposable such that G is not reductive. More specifically 
let £ and be direct sums 



£= ® Mi 

l<i<r 



I £.} and JF 



KKs 



where Mj and A*"; are finite dimensional vector spaces and £i, Ti are simple sheaves, i.e. 
their only endomorphisms are the homotheties, and such that Hom(£^j,£^j) = for i> j 
and Yk)m{Ti^Tm) = for I > m. In this case we call homomorphisms £ ^ oi type 
(r, s). Then the groups Aut{£) and Aut(JF) can be viewed as groups of matrices of the 
following type. The group Aut(£^), say, is the group of matrices 



/ gi 

U2l 92 
\ Url ■■ ■ 



Ur 



\ 



-1 9r J 



where Qi G GL(Mj) and Uji G Hom(Mj,M,- ® }iom{£i, £j)). 

In the literature on moduli of vector bundles and coherent sheaves many quotients of 
spaces FB.om{£,J^) of type (1,1) by the reductive group Aut(£^) x Aut(JF) have been 
investigated, see for example |0, |T^, |T^, [^, ^6|. The moduli spaces described in this 
way are the simplest ones, and this allows to test in these cases some conjectures that are 
expected to be true on more general moduli spaces of sheaves (cf. 0, |3^). We think 
that the moduli spaces of morphisms of type (r, s) will be as useful to treat other less 
simple moduli problems of sheaves. In fact, if one wants to use the spaces Hom(£^,jF) 
as parameter spaces for moduli spaces of sheaves, which are as close as possible to the 
moduli spaces, the higher types (r, s) are unavoidable. 

The homomorphisms in a Beilinson complex of a bundle on projective ?7,-space, for exam- 
ple, have in general arbitrary type (r, s) depending on the dimensions of the cohomology 
spaces of the bundle. In several papers, see fS^] for example, semi-stable sheaves or 
ideal sheaves of subschemes of projective spaces, are represented as quotients of injective 
morphisms of type (r, s), and one should expect that the moduli spaces of such sheaves 
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are isomorphic to a good quotient of an open subset of the corresponding space of homo- 
morphisms. In some cases of type (2, 1) this has been verified for semi-stable sheaves on 
P2 in 0. 

In case of type (r, s) there are good and projective quotients if one restricts the action to 
the reductive subgroup 



Gred = ll GL(M,) X J]GL(iVO. 



This has been shown recently by A. King in |^T|. The quotient problem for Hom(£^,jF) 
of type (r, s) with respect to the full group Aut{£) x Aut(jF) is however the generic one 
and indispensable. 

Unfortunately the by now standard geometric invariant theory (GIT) doesn't provide a 
direct answer for these quotient problems in case Aut(£^) x Aut(JF) is not reductive. There 



are several papers dealing with the action of an arbitrary algebraic group like [|T6[, [0, 0], 
Q and older ones, but their results are insufficient for the above problem. The conditions 
of [|TB| are close to what we need, but they don't allow a concrete description of the set 



of semi-stable points in our case and they don't guarantee good or projective quotients. 



see remark 4.1.2. 



1.2. The main idea 

Our procedure is very close to standard GIT and we finally reduce the problem of the 
quotient to the one of a reductive group action. We introduce polarizations A e Q''+'' of 
tuples of rational numbers for the action of G on the affine space in analogy to the ones 
of A. King in which are refinements of the polarizations by ample line bundles on 
the projective space FW, and then introduce open sets W^i^GjA) C W'^'^{G,A) of stable 
and semi-stable points depending on A and study the quotient problem for these open 
subsets. There are chambers in Q''"'"* such that the polarizations in one chamber define 
the same open set, in accordance with the chamber structure in Neron-Severi spaces of 
polarizations in the reductive case, see f.e. p I. Dolgachev - Y. Hu and M. Thaddeus. 
However, in contrast to the reductive case, good quotients W'^'^^G, A)/ /G don't exist for 
all polarizations, see ^4.2| . As a main achievement we are providing numerical conditions on 
the polarizations, depending on the dimensions of the spaces Mj and Ni, under which such 
quotients exist. The main step for that is to embed the group actions G x W —>■ W into 
an action G x W —>■ W of a reductive group G and to compare the open sets W^'^{G, A) 
and W'^'*(G, A), where A is a polarization for the G-action associated to A. 



1.3. Construction of quotients by non reductive groups. 

To be more precise, a polarization A is a tuple (Ai, ... , A^, /^i, . . . , fis) of positive rational 
numbers, called weights of the factors Mj Si and Ni ^ J-'i respectively, which satisfy 
XiiTLi = f^i^i = 1) where nii^ni denote the dimensions of the spaces of the same 
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name. We use then the numerical criterion of A. King, as definition for semi- 

stabihty with respect to the reductive group Gred- An element w E W is {Gred, A)-stable 
if for any proper choice of subspaces C Mi, N'l C Ni such that w maps ©(M^' ® £i) 
into (B{N'i J^i), we have ^ A^m^ < fiin'i, or semi-stable if equality is allowed. Let 
W'^{Gred, C iy**(Gred, A) dcuotc the sct of stablc and semi-stable points so defined. 
If H G G is the unipotent radical of G, which is generated by the homomorphisms 
Si £j and JF^ — > JF^ for i < j and I < m, we say that w is (G, A)-(semi-)stable 
if h.w is {Gred, A )-(semi-) stable for any h E H, see ^A\ . We thus have open subsets 
W{G,A) C W{Gred,A) and W'{G,A) C W{Gred,A)- 

The main result of our paper is that there are sufficient numerical and effective bounds 
for the polarizations A such that W^^{G,A) admits a good and even projective quotient 
W^^{G, A)/ /G and that in addition iy**(G',A) admits a geometric quotient, which is 
smooth and quasi-projective, see proposition \6.1.1\ and the results \7.2.3i , \7. 5. Sj , and section 



The definitions of good and geometric quotients are recalled in |6.1| . By using correspon- 
dences between spaces of morphisms, called mutations, it is possible to deduce from our 



results other polarizations such that there exists a good projective quotient (see []T0[, [|T2])- 

All this is achieved by embedding the action G x W ^ W into an action G x W —>■ W 
of a reductive group and then imposing conditions for the equality W^'^{G,A) = W (1 
W^^{G, A), where A is the associated polarization. The quotient is then the quotient of 
the saturated subvariety GW'^'^{G, A) C W'^*(G,A). The quotient will be projective if 
G.W \ G.W doesn't meet W'^*(G,A). Also for this, numerical conditions can be found 
in section 8. 

The idea of embedding the non-reductive action GxW ^ W into the action G x W W 
is simply to replace the £i by £i using the evaluation maps B.om{£i, Si) — Si. It is 
explained in and |5.1.1| that this is the outcome when we start to replace the sheaves Si 
step by step and similarly for the sheaves JF;. Since we have to deal everywhere with the 
dimensions of the vector spaces Hom(^j, Sj) and Hom(jF;, JF^) which form the components 
of the unipotent group H, we have translated the whole setup into an abstract multilinear 
setting and related actions by technical reasons. This gives more general results although 
we have only applications in the theory of sheaves. The reader should always keep in 
mind the motivation in 

The results obtained in the simplest case (morphisms of type (2,1) or (1,2)) are stated in 
|1.5| . They are characteristic for the general case in which only the conditions are more 
complicated. 



1.4. Remark on finite generatedness 

One would expect that the quotients of W could be obtained by first forming the quotient 
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W/ H with respect to the unipotent radical H and then in a second step a quotient of W/ H 
by G/H = Gred- However, the actions of unipotent groups behave generally very badly, 
||T9| , and we are not able to prove that the algebra C[l^/if] is finitely generated. This 
would be an essential step in a direct construction of the quotient. Of course, the main 
difficulty also in this paper arises from the presence of the group H. The counterexample 



of M.Nagata, ||2^, also shows that the finite generatedness depends on the dimensions of 
the problem. So from a philosophical point of view we are determining bounds for the 
dimensions involved under which we can expect local affine G-invariant coordinate rings 
which are finitely generated, and thus to obtain good quasi-projective quotients, even 



so the bounds might not be the best. The simple examples 4.2, [4.3| show that a good 



quotient A)/ /G might not exist if the conditions are not fulfilled. 



1.5. MORPHISMS OF TYPE (2,1) 

In this case the homomorphisms of sheaves are of the type 

mi£i © 171282 — > niTi, 

where we use the notation mS for C™ ® S. For this type a polarization is given by a pair 
(Ai, A2) of positive rational numbers such that Aimi + \2m2 = 1. It is determined by the 
rational number t = m2\2 which lies in [0, 1]. Writing W^^{t) for W^^ and W^{t) for 
for the moment, our results depend on constants c{k) defined as follows : Let 

r : Hom(^i, J^i)* ® Hom(^i,^2) — > Hom(^2,^i)* 

be the linear map induced by the composition map Hom(£2,-^i) ® Hom(£^i,£^2) 
Hom(£^i, J^i) , and 

r;. = r (g) Jcfc : Hom(^i, J^i)* ® (Hom(£i, £2) ® C^) — > Hom(f2, J^iY ® 

Let /C be the set of proper linear subspaces K C Hom(£^i,£^2) ® C*^ such that for every 
proper linear subspace F C C'^, K is not contained in Hom(£^i,£^2) ® F. Let 

_ ^ codim(rfc(Hom(gi, J^i)* OiT) ^ 

C[K,) — SUp( — — — ). 

K&K. codim(if) 

1.5.1. Theorem: There exists a good projective quotient W'^^{t)//G and a geometric 
quotient W'{t)/G if 

m2dim(Hom(^i,^2)) , + ^ ^- m (p p\\ ( ^"^2 

t > and t > dim(Hom(£:i,£:2)).c(m2) — . 

dim(Hom(ci, 02)) + rni rii 

In the case of morphisms mi(9(— 2)©m2(9(— 1) — > uiO on projective spaces the constants 
have been computed in and we obtain the more explicit result : 
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1.5.2. Theorem: Let n > 2 be an integer. There exists a good projective quotient 
W'^^{t)//G and a geometric quotient W''^{t)/G in the case of morphisms 
mi (9 (—2) © m20{—l) — > uiO on the projective space P„ if 

{n + l)m2 



t > 



{n + l)m2 + mi 



in + l)mn(m2 — 1) 
2ni[m2[n + 1) - 1) 

in + l)^m2 

^ > o( Jo\ > n + 1. 

2(n + 2)ni 



1.6. Construction of fine moduli spaces of torsion free sheaves 



In section we construct smooth projective fine moduh spaces of torsion free coherent 
sheaves on P„ using morphisms 

(*) C(-2)®C2 ^C(-1)©(C®C'=), 

(for {n + l){n + 2) / 2 < k < (n + 1)^ ). More precisely we prove that for all polarizations, 
semi-stable morphisms are injective outside a closed subvariety of codimension > 2, hence 
their cokernels are torsion free sheaves. A generic morphism is injective and its cokernel 
is locally free. In this case we can construct 

(n + l)(n + 2) 



n + k + l 



distinct smooth projective moduli spaces Mi, ■ ■ ■ ,Mq of such morphisms, of dimension 
2{n — 1) + k{{n + 1)^ — k). Moreover, all the Mj are birational to each other. For 1 < 
i < q, we construct a coherent sheaf Si on Mj x P„, fiat over Mj, such that for every closed 
point z G Mj, Siz is isomorphic to the cokernel of the morphism (*) corresponding to z. 
We prove that Mj is a fine moduli space of torsion free sheaves with universal sheaf Si. 
In particular, this means that for every closed point z G Mj, the Kodai'ra-Spencer map 

TMi^'Ext\Si,,Si,) 

is bijective, and for any two distinct closed points zi,Z2 G Mj, the sheaves Siz-^, Siz^ are 
not isomorphic. 



1.7. Open problems 

Even in the simplest case of morphisms of type (2,1) we do not know what all the polariza- 
tions are for which a good quotient W'^'^ / /G exists. More generally it would be interesting 
to find all the saturated open subsets U oiW such that a good quotient (quasiprojective 
or not) U / /G exists, or all the open subsets U such that a geometric quotient U /G exists. 
The corresponding problem for reductive groups has been studied in 1.12, 1.13, and 
in m, 01. 
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1.8. Organization of the paper 

In section 2 we describe our problem in terms of multilinear algebra. 



In section 3 we recall results of A. King, |21|. The reductive group actions considered in 



this paper, the action of Gred on W and that of G on W, are particular cases of [^. We 
also discuss the relation of A-(semi-) stability in W with that in the projective space FW. 
But we cannot work solely on the projective niveau, because the embedding W C W is 
not linear. 

After defining G-(semi-) stability for the non-reductive group in section 4 we describe the 
embedding in section 5 and introduce the associated polarizations. 

Section 6 contains the step of constructing the quotient W'^'^{G, A)/ /G using the GIT- 
quotient W"^(G,A)//G of A. King. 

Sections 7 and 8 are the hard parts of the paper. Here the conditions of the weights 
which define good polarizations are derived. It seems that the constants appearing in 
these estimates had not been considered before. 

In section 9 we are investigating a few examples in order to test the strength of the 
bounds. Here we restrict ourselves to small type (2, 1), (2, 2), (3, 1) in order to avoid long 
computations of the constants which give the bounds for the polarizations. What we 
discover in varying the polarizations are flips between the moduli spaces, as one has to 
expect from the general results on the variation of linearizations of group actions, cf . P2] , 



0, In example |9]^ we have a very simple effect of a flip, but in example |9]^ the 



chambers of the polarizations look already very complicated. 

In section 10 we define new fine moduli spaces of torsion free sheaves using our moduli 
spaces of morphisms. 

Acknowledgement. The work on this paper was supported by DFG. The first author 
wishes to thank the University of Kaiserslautern, where the work was started, for its 
hospitality. 



2. The moduli problem for decomposable homomorphisms 

Let £ = Q) Si ^ Mi and J-" = Q) J-'i ^ Ni he semi-simple sheaves as in the introduction. 
In order to describe the action of G = Aut(£^) x Aut(jF) on W = B.om.{S , J-') in greater 
detail we use the abbreviations 

Hii = Rom{Si,J^i) 
Aji = }iom{Si,Sj) 
Bml = Hom(:F;,jr^) , 
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such that we are given the natural pairings 



Hij i 


5 - 


Hii 


for 


i < 


3 




5 - 




for 


i < 


3<k 


B,nl d 


5 Hii - 


~^ Hmi 


for 


I < 


m 




5 Bjnl - 


Bnl 


for 


I < 


m < n 



The group G consists now of pairs {g,h) of matrices 



9 = 



/ 9. 





92 



Ur 



\ 



-1 9r / 



and h — 



( h 

V21 





h2 



Vs^s-l IT'S / 



\ 



with diagonal elements Qi G GL(Mi), hi G GL(A''/) and Uji G Hom(Mi, Mj Aji), 
Vml G Hom(iVi, Nm (S) Bml)- 

Similarly a homomorphism w G Hom(£^,jF) is represented by a matrix w = {(fu) of ho- 
momorphisms ipu G Hom(Mj, Ni ® Hn) = Hom(i7;* ® Mj, A^";). Using the natural pairings, 
the left action (^f, h).w — hwg"^ of G on is described by the matrix product 



/ hi 

V21 
\ Vsl 





h2 



\ 



Vs,s-1 




he 



( 



) 



\ 



( 



^sr J 



9i 

U21 





92 



\ Url 



— 1 



\ 



where the compositions v^i o and (pij o uji are compositions as sheaf homomorphisms 
but can also be interpreted as compositions induced by the pairings of the vector spaces 
above. Thus the group G, the space W and the action are already determined by the 
vector spaces Aji, Bmi, Hu and the pairings between them. Therefore, in the following we 
define G, W and the actions GxW hy abstractly given vector spaces and pairings. 

The resulting statements can then be applied to systems of sheaves by specifying the 
spaces as spaces of homomorphisms as above. 



2.1. The abstract setting 

Let r, s be positive integers and let for l<i<j<r, 1 < I < m < s finite dimensional 
vector spaces Aji, Bmu Hu be given, where we assume that A^ = C and Bu = C. Moreover 
we suppose that we are given linear maps, called compositions, 

Hij (g) Aji ^ Hu for I <i < j < r, l<l < s 

Akj Aji — > Aki for l<i<j<k<r 

Bmi <8) Hu Hmi for 1 < i < r, 1 < Z < m < s 

Bnm ® Bmi B^i for 1 <l <m <n < s. 
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We assume that all these maps and the induced maps 

HI, ® A^, ^ HI. and H*^, ® B^i ^ H* 

are surjective. This is the case when all the spaces are spaces of sheaf homomorphisms 
as above for which the sheaves Si and are line bundles on a projective space or each of 
them is a bundle 0,P{p). 

We may and do assume that these pairings are the identities Hi — j, I — m etc. . Finally, 
we suppose that these maps verify the natural associative properties of compositions. This 
means that the induced diagrams 

Akj ® Aji ® Aih > Aki (g) Aih Bon ^ ^ B^i ^ S,,^ ® B^i 

., " '1 

Akj ® Ajh ^ Akh Bon ® Bni > Bol 

Hik ® Akj O Aji > Hij ® Aji Bnm O B^i ® Hu ^ Bni ® Hu 

,, I I 

Hlk <8) Aki ^ Hu Bnm ® Hjni ^ Hni 

Bmi <S) Hij ® Aji > Hjnj <S) Aji 

Bml ® Hu ^ Hjni 

are commutative for all possible combinations of indices. 

In our setup we also let finite dimensional vector spaces Mj for 1 < i < r and Ni for 
1 < Z < s be given and we consider finally the vector space 

W ^® Hom(Mi, Ni Hu) = © Hom(//* ® Mi, Ni) 

i,l i,l 

where summation is over 1 < i < r and 1 < / < s. This is the space of homomorphisms in 
the abstract setting. The group G and its action on W are now also given in the abstract 
setting as follows. 
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2.2. The group G 

We define G as a product GlX Gr of two groups where the left group Gl replaces Aut{£) 
and the right group Gr replaces Aut(.F) in our motivation. Let Gl be the set of matrices 

(91 ••• \ 

U21 92 '■ 
': ■•• ■•. 

\Uri ■■■ Ur,r-1 9r J 

with Qi e GL(Mi) and uji G Hom(Mi,Mj Aji) = Rom{A*i (g) Mi,Mj). The group law 
in is now defined as matrix multiplication where we define the compositions Ukj * Uji 
naturally according to the given pairings as the composition 

Mi Mj ® Aji — — > Mk Akj Aji ^ Mfe ® Aki. 

Explicitly, if g has the entries gf,, Uji and g' has the entries g'^, u'j^ then the product 

/ = 9' 9 

in Gl is defined as the matrix with the entries g" — g[o g^^ in the diagonal and 

u'L = u'ki °9i+ ^ u'kj * + {9'k ® id) o Uki 

i<j<k 

ior 1 < i < k < r. The verification that this defines a group structure on Gl is now 
straightforward. 

As a set Gl is the product of all the GL(Mj) and all Hom(Mj, Mj ® Aji) for i < j and 
thus has the structure of an affine variety. Since multiplication is composed by a system 
of bilinear maps it is a morphism of affine varieties. Hence G^, is naturally endowed with 

the structure of an algebraic group. The group Gr is now defined in the same way by 
replacing the spaces Mj and Aji by Ni and Bj^i- Finally G = Gl x Gr is defined as an 
algebraic group. 



2.3. The action of G and W 

We will define a left action of Gr and a right action of Gl on such that the action of G 
on W can be defined by {g, h).w = h.w.g~^. Both actions are defined as matrix products 
as described above in the case of sheaf homomorphisms using the abstract compositions 
as in the definition of the group law. 

If w has the entries (pu e Hom(if*- ® Mi,Ni) and g & Gl has the entries gi and Uij then 
w.g is defined as the matrix product 
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I 9i 

U21 92 



\ 







\Url ■■■ Ur,r-1 Qr ) 



with 



° 9i + ^ij * 'iJ'ji {Hi = r the last sum is 0) , 



i<j 



where ipij * Uji is the composition 

Mi Mj Aji ^ Ni® Hij Aji ^ Ni^ Hu 
or dually the composition 

H;, ®Mi^ H*j (8) A*i ®Mi^ H*j Mj Ni. 

The left action of Gr is defined in the same way. In the next two sections we give 
an analysis of stabihty and semi-stability for the action of G and its natural reductive 
subgroup Gred- In the reductive case this is due to A. King. 



2.4. Canonical subgroups of G 

We let Hl <Z Gl and Hr C Gr be the maximal normal unipotent subgroups of and 
Gr defined by the condition that all Qi — idu^ and all hi — id^i- Then H — Hl x Hr is a 
maximal normal unipotent subgroup of G. Similarly we consider the reductive subgroups 
GL,red and Gfi^red of Gl and Gr defined by the conditions Uji = and v^i = for all 
indices. Then Gred = GL,red x GR^red is a reductive subgroup of G and it is easy to see that 
G/H ^ Gred- The restricted action of Gred is much simpler and reduces to the natural 
actions of GL(Mi) on Mj and GL(iV;) on Ni 
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3. Actions of reductive groups 
3.1. Results of A. King 

Let Q be a finite set, T C Q x Q a subset such that the union of the images of the two 
projections of F is Q. For each a E Q, let m„ be a positive integer, a vector space 
of dimension and for each G F, let Vap be a finite dimensional nonzero vector 

space. Let 

Wo= © Hom(M« ® Vaf3, Mp). 
(a,/3)er 

On Wq we have the following action of the reductive group 

Go = n GL(Ma) 

arising naturally in this situation. If (//3a) G Wq and {go) G Gq; then 

i9a)-ifl3a) = (9/3 ° ffSa ° {.Qa ® id)'^). 

Let (ea)aGQ be a sequence of integers such that 

e„ma = 0. 

To this sequence is associated the character x of Gq defined by 

This character is trivial on the canonical subgroup of Go isomorphic to C* (for every 
A G C*, the element {go) of Gq corresponding to A is such that ga = X-id for each a). 
This subgroup acts trivially on Wq. A point x G Wq is called xsemi-stable if there exists 
an integer n > 1 and a polynomial / G C[11/o] which is x"-invariant and such that f{x) 7^ 
(/ is called x^-invariant if for every w G Wq and 5^ G Gq we have f{gw) = X^ {q) f i.'^)) ■ 
The point x is called x-sta6/e if moreover 

dim(Gox) = dim(Go/C*) and if the action of Gq on {w G Wq, f{w) 7^ 0} is closed. 

A. King proves in the following results : 

(1) A point X = {f/3a) G Wq is x-semi-stable (resp. x-stable) if and only if for each 
family (M^), a G Q, of subspaces C which is neither the trivial family (0) 
nor the given family (Mq) and which satisfies 

for each (a,/3) G F, we have 

^e„dim(M^) < (resp. < ). 
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(2) Let Wq'^ (resp. Wq) be the open subset of Wq consisting of semi-stable (resp stable) 
points. Then there exists a good quotient 

TT : W^' — > M 

by Gq/C* which is a projective variety. 

(3) The restriction of this quotient 

— > M' = 7r{W^) 
is a geometric quotient and M'* is smooth. 



3.2. Polarizations 

The (semi-)stable points of Wq remain the same if we replace (cq) by (cCq), c being 
a positive integer. So the notion of (semi-)stability is fully described by the reduced 
parameters (^), where 

t ^ ^ dcJTla- 

a£Q,ec,>0 

So we can define the polarization of the action of Gq on Wq by any sequence {ca)a&Q of 
nonzero rational numbers such that 

Cairia = , ^ CaTTla = 1. 

By multiplying this sequence by the smallest common denominator of the Ca we obtain 
a sequence (cq.) of integers and the corresponding character of Gq. Therefore the loci of 
stable and semi-stable points of Wq with respect to Go and a polarization Aq = ( 
well defined and denoted by 

l^o'(Go,Ao) and W^'{Go,Ao). 



3.3. Conditions imposed by the non-emptiness of the quotient 

If Wq is not empty, the must satisfy some conditions. We will derive this only in 
the three situations which occur in this paper. Polarizations satisfying these necessary 
conditions will be called proper. The first is that of the action of Gred in |2.4| and the 
second is that of G and W in section 5, and the third is the case in between occurring in 



3.3.1. First case 

Let r, s be positive integers. We take 

Q = {ai.. . ,ar,Pi,... , Ps}, r = {ai . . . , Or} x {/3i, . . . , Ps}- 
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This is the case of morphisms of type (r, s). For 1 < z < r, let M^. = M^. if e^j > 0, 
and {0} otherwise, and for 1 < Z < s, let M^^ = Mp^. Then if one e^. is not positive, we 
have 

J2eadim{M'J > 
aeQ 

and (M^) (Mq), so in this case no point of Wq is stable. So we obtain , if Wq is 
non-empty, the conditions 

e„. > , for any i, and e^^ < , for any I. 

A proper polarization is in this case a sequence (Ai, ... , A^, — /^i, • • • , — A*s) of rational 
numbers such that the Aj and the are positive and satisfy 

l<i<r l<l<s 

3.3.2. Second case 

This case appears when we use a bigger reductive group to define the quotient (this is the 
case of W later on). Let r, s be positive integers. Here we take 

(5 = {ai . . . , a^, . . . , /3J, T = {{a^, Q;j_i), 2 < i < r, (cvi, (A, A-i), 2 < i < s}. 
Then the necessary conditions for Wq to be non-empty are: 

CajTaaj > for any i, and ^I3i''^i3i < for any m. 

i<j<r l<l<m 

To derive the first set of conditions we consider for any i the family (M^) for which 
M^. = if i < j < r and = for all other 7 G Q. Then /^/^(M; ® K^j) C for 
any / e Wq and any (a, /3) e F. If / is stable we obtain 

- Y '''"j^S = ^e^dim(M^) < 

i<j<r jeQ 

Moreover, if the family (A£ ) is defined by M'^. = for 1 < j < r, M'^^ — Oifm<l<s 
and = My else, we obtain directly 

l<Z<m 7eQ 

A proper polarization in this case is then a sequence (pi, . . . , Pr, —ci, ■ ■ ■ , —Cs) of rational 
numbers satisfying 

l<i<r l<l<s 

and 

PjTriaj > for any i and cr^m^j > for any m. 

i<j<r l<l<m 
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We could also drop the normalization condition. 
3.3.3. Third case 

This case is a combination of the first and second case. It appears in the proof of the 
equivalence of semi-stability in |7.3| . Here Q is the same as in the previous cases and 

r = {{ai, , 2<i<r , {ai, A) , 1 < / < s}. 

Now the necessary conditions for Wq to be non-empty are: 

CajTria^ > for any i, and ep^ < for any /. 

i<j<r 

The first condition follows as in the second case when we consider the family (M^) with 
M^. = for i < j < r and = for all other 7 G Q. The second condition follows 
when all are zero except M^^ = Afg^ for one /. Again a proper polarization in this case 
is a sequence (pi, ... , pr, —pi, ■ ■ ■ , —pi) with 

l<j<r l<l.<s 

and 

Pjnia. > for any i and pi > for any I. 

i<j<r 

3.4. The action of Gq on F{Wo) 

We suppose that we are in one of the first two preceding cases and that there exist stable 
points in Wq. Let P be a nonzero homogeneous polynomial, x'^-invariant for some positive 
integer n. The x"-invariance implies that P has degree n.t where in case 1 (action of Gred 
on W) 

l<i<r 

and in case 2 (action of G on W) 

l<i<r l<Ks 

To see this let A G C* and let g be given by = X^^id and Qf^^ = id in the first case and 
by = X~Hd and g^^ = X^~'^id in the second case. Then gx = Xx and = -^"^ 

both cases, such that P{Xx) = A"*P(x). 
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Now we will see that there exists a Go-hne bundle L on P(lVo) such that the set W^^ 
of semi-stable points is exactly the set of points over ^{Wi^'^'^iCji^, C), which is the set of 
semi-stable points in the sense of Geometric Invariant Theory corresponding to 

1^ = O^wo){t), 



cf. ||2^, |2^, [0. Here the action of Go on C is the natural action multiplied by x- More 
precisely, the action of Go on Wq induces an action of this group on S^Wq and on S^Wq 
by: 

{g.F)iw) = Fig-'w) 

for aA\ g E Go, w E Wq and F G S^Wq, viewed as an homogeneous polynomial of degree 
t on Wq. The line bundle space L of £ is acted on by Gq in the same way : if ^ G L^^^ 
then g.^eL^g^y is the form on < gw >*'^^= L^g^^ given by (g-Oiv) = ^ia'^v)- 
modify now the action of Gq on L (resp. S^Wq) by multiplying with xio) '■ 



9*i = x{g)9-i for eGL<^>, oi g * F = x{g)g.F ioi F e H\nW,)X) = S'W, 



• 



Now P G H^{¥{Wq), C®^) is an invariant section if and only if P is a homogeneous 
polynomial of degree tn which satisfies 

From the definition of semi-stable points in Wq and P(PVo) with respect to the modified 
Go-structure on £ = (9p(vvo)(^), we get immediately 



3.4.1. Lemma: Assume that iyQ*(Go, Aq) 7^ and let t be defined as above in the two 
cases ofWo. Then the set IVq''(Go,Ao) is the cone of the set F(WoY''{Go,Of>(Wo)(t)) as 
defined in G.I. T. 



There are two definitions of stable points in P(iyo); the classical one, given in [^, ||2^ 



and a more recent one, given in [^]. If we take D. Mumford's definition, the cone of the 
set of stable points in P(Wo) does not coincide with Wq because every point of P(PVo) 
has a stabilizer of positive dimension. In fact there is a subgroup of Go/C* of positive 
dimension which acts trivially on P(VFo)- Iii the first case for example such a group is given 
by goi = Aic? and g^^ = fiid with A,/i G C*. If we want to keep the coincidence between 
the sets of stable points for one and the same group, we would have to consider the action 
of a smaller reductive group in order to eliminate additional stabilizers. We will do this in 
|3.5| only in the first case. If we take the definition of V.L. Popov and E.G. Vinberg, then 
we obtain that the set iyQ(Go, Ao) is exactly the cone of the set P(PFo)'*(Go, Op(iyo)(^)) 
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3.5. The group G' 

Let G and W be as in section 2 and let A = (Ai, ... , A^, — /ii, . . . , — /^s) be a proper 



polarization as in |3.3.1| for the action of Gred on W . It is then convenient to use the 
subgroup G^g^ of Gred consisting of elements {{gi), {hi)) satisfying 

JJ det{gi)'''' = Yl det{hi)^'' = 1, where aji = dim{Aji) and b^i = dim(E^z). 



l<i<r KKs 



We consider the action of G^^^ on C induced by the modified x-action of Gred- Now the 
set W^^Gred, A) of x-stable points of W is exactly the cone over the locus F(W)^{G'j.^^, C) 
of stable points of P(iy) in the sense of Geometric Invariant Theory. 



4. Semi-stability in the non-reductive case 

Let G and W be as in section 2. A character x on Gred as in King's setup can be extended 
to a character of G. Also the modified action of Gred on C can be extended to an action 
of G. Let G' be the subgroup of G defined by the same equations as for G[^^. It contains 
H and G'.^^, and we have G' / H G'^ed- 

In the case of the action of Gred on W a. proper polarization is given by a sequence 
Ai, . . . , Ar, /ii, . . . , /^s of positive rational numbers such that 

= ^ fiiui = 1. 

l<i<r l<l<s 

More precisely, the polarization is exactly the sequence (Ai, ... , A^, — /^i, . . . , — /^s)- The 
parameter Aj (resp. fii) will be called the weight of the vector space Mj (resp. Ni). We 
see that the dimension of the set of possible proper polarizations is r + s — 2. Let t denote 
the smallest common denominator of the numbers Aj and /i/ and x the character of Gred 
defined by the sequence of integers (— tAi, . . . , —tXr, t/ii, . . . , t/is). Let 

C = Cp(iy)(t) with t = rUitXi. 

l<i<r 

As we have seen, if we consider the modified action of Gred on C, then the x-semi- 
stable points of W are exactly those over the semi-stable points of P(iy) in the sense of 
Geometric Invariant Theory with respect to the action of G^ed/C* on C. The x*"^- invariant 
polynomials are the Gred-invariant sections of 

We are now going to define a notion of (semi-) stability for the points of W with respect 
to the given action of the non-reductive group G. Let if C G be the above unipotent 



group, see also 
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4.1. Definition: A point w ^ W is called G-semi-stahle (resp. G-stable) with respect 
to the (proper) polarization A = (Ai, ... , A^, — yUi, . . . , — /^s) if every point of Hw is Gred^ 
semi-stable (resp. Gred^stable) with respect to this polarization. 

We denote these sets by W'"'{G,A) resp. W''{G,A). 

For many of the quotient problems for the spaces of homomorphisms between (BrriiSi and 
(BrijJ^j and their cokernel sheaves this is a fruitful notion. In ^]2| we investigate an example 
with an explicit description of the open sets A) C H^**(G,.ed, A). This example also 

shows that the existence of a good quotient depends on the choice of the polarization. 



4.1.1. Situation for type (2,1): 

In the case of morphisms of type (2, 1) we have /ii = 1/ni and the polarization is com- 
pletely described by the single parameter t = m2X2- We must have < t < 1. 

A polarization such that there exists integers m[, m'2, n[, with < n[ < ni, < m'^ < rrii, 
such that m[ni — min[, m^Ui — m2n[ are not both 0, and that 

Aim, + X2mL = — 

is called singular. There are only finitely many singular polarizations, corresponding to 
the values Q < ti < t2 < ■ ■ ■ < tp < 1 of t. Let to = 0, tp+i = 1. If A, A' are 
polarizations corresponding to parameters t, t' such that for some i G {0, ■ ■ ■ ,p} we have 
ti < t,t' < tj+i, then 

W''{G, A) = W''{G, A') and W'{G, A) = W'{G, A'). 

Hence there are exactly 2p + 1 notions of G- (semi-) stability in this case. Moreover, 
if mi, m2 and ni are relatively prime, and A is a non singular polarization, we have 
W''{G,A) = W'{G,A). 

In the general case of morphisms of type (r, s), it is not difficult to see that there are only 
finitely many notions of G- (semi-) stability. 



4.1.2. Remark: In semi-stability is defined as follows : A point w G W is semi-stable 
if there exists a positive integer k and a G'-invariant section s of £^ such that s{w) 7^ 
(there is also a condition on the action of H). It is clear that a semi-stable point in the 
sense of Fauntleroy is also G-semi-stable with respect to (Ai, ... , A^, — /ii, • . . , — yU^)- It 
is proved in [|16| that there exists a categorical quotient of the open subset of semi-stable 
points in the sense of |16|, but it is not clear that all G-semi-stable points are semi-stable. 
Moreover, in the general situation of [0 there is no way to impose conditions which 
would imply that the categorical quotient is a good quotient or even projective. Using 
definition we are able to derive a criterion for the existence of a good and projective 
quotient of W under the action of G. 
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4.2. Existence and non-existence of good quotients, an example 

We show here that we cannot expect that a good quotient W^'^i^K.G) / /G will exist for 
any polarization A. 

We consider morphisms 20{—2) 0{—l) ® O on P2. There are 3 notions of 
G- (semi-) stability in this case, two corresponding to non singular polarizations. For one 
of the non singular polarizations the quotient W^{K,G) / G exists and for the other we 
prove the inexistence of a good quotient W^{K, G)/ /G. 

Let y be a complex vector space of dimension 3, and P2 = PV^. Let 
W = Hom(2C(-2), C(-l) © O) on P2. A polarization for the action of G on Vl^ is a 
triple (1/2, — /xi, —1^2) of positive rational numbers such that /ii + /i2 = 1- As in [4.1.1| a 
polarization depends only on /ii. There is only one singular polarization, corresponding to 
/ii = 1/2. Hence if we consider only non singular polarizations there are only two notions 
of G- (semi-) stability, the first one corresponding to polarizations such that fii > 1/2 and 
the second to polarizations such that /xi < 1/2. In both cases semi-stable points are 
already stable. We are going to show that in the first case W^{G,A) has a geometric 
quotient which is projective and smooth and that in the second case iy^(G, A) doesn't 
even admit a good quotient. 

The elements x & W and g E G are written as matrices 

( zi Z2\ , f f a 

x = [ and g = [a,[ 

where Zi, Z2 E V*, qi, q2 € S'^V*, a G GL(2), a,peC* and z G V*. 
4.2.1. The case yUi > 1/2 

In this case W'^{G,A) has a geometric quotient which is the universal cubic Z C WV x 
FS^V* of the Hilbert scheme of plane cubic curves in P2 = FV. The quotient map is 
given by a; ((zi A Z2), (^1^2 - Z2qi)) 

Remark If > 3/4, then /ii > 3/i2 and the conditions of |1.5.1| (in the dual case (1, 2))for 



a good and projective quotient to exist in this case are satisfied. 
The proof is done in several steps. 

(1) claim 1: Let x G be as above. Then 

(i) X G W^^Gred, A) if and only ii zi A Z2 ^ in A'^V* and qi, q2 are not both zero. 

(ii) X G WiG, A) if and only ii zi A Z2 and det(x) = 2:1^2 - -22^1 7^ in S^V*. 



Proof of claim 1. (i) follows easily from the criterion (1) in |3.1| . As for (ii) let x G 
W'{Gred,A) with det(x) ^ 0. Then det(/i.x) = det(a;) 7^ for any /i = ( ^ ? ) which 
implies that also h.x G W^iGred, A). Let conversely x G ^^'^(G, A). Then det(x) 7^ 
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because otherwise there is a hnear form z G V* with qi = zz\ and q2 = zz2 and with 
h = the element h.x is the matrix ( ) which is not in W^{Gred, A). 

(2) By the resuh of A. King in |3TT| , (3), there is a geometric quotient {Gred, / Gred 
which is smooth and projective. 

claim 2: W'{Gred,A)/Gred = F(g* ® S^V*). Here Q* = is the dual of the tau- 

tological quotient bundle over PV^. (The dimension of this quotient variety is 13 while 
dimW = 18 and dimGred/C* = 5). 

To verify claim 2 we consider the map 

X = ( ^2 I A ((2:1 A 2:2), (2:1 ® g2 - ^2 ® gi)) 

from W'iGred, A) to ¥V x F{V* O S^V*) C ¥{V* (g) S^V* O Opy) where we identify PA V* 
with FV via {Z1AZ2) ^ (a), 2;i(a) = 2:2(0) = 0. Then each a{x) G ¥{Q*^-^(gS'^V*) because 
Q*^'^ C V* is the subspace of forms vanishing in (a). It follows immediately that a is a 
morphism 

W''{Gred,A)^nQ*^S^V*) 

which is surjective and Gred^equivariant. It induces a morphism of the geometric quotient 
to F{Q* (g) S^V*) which is even bijective. Since both, the quotient and the target are 
smooth, this is an isomorphism. 

(3) Since Q* C V* ^ O^y we have an induced homomorphism Q* ® S'^V* S^V* ® O^v- 
It is the middle part of the canonical exact sequence 

^ k^Q* ®V* ^Q*® S^V* 5V* ® Opv ^ Cpi/(3) ^ 

of vector bundles on PV^. Let Z be the kernel of ev. From the left part of the sequence 
we obtain the affine bundle 

P(Q* ® 5V*) \ P(A2Q* ® V*) ^ P(Z) c PV" X PSV*. 

Here P(2) = Z is nothing but the universal cubic and the fibres of (3 are isomorphic to 
V*. 

claim 3: W'{G, A) C W'{Gred, A) is the inverse image of F{Q* ® S^V*) \ P(A2g* ® V*) 
under a and aiwiCA) ^ geometric quotient with respect to Gred- 

Proof of claim 3. Zi ® q2 — Z2 ® qi belongs to A^Q^^^ ® V* if and only if Ziq2 — Z2qi = 0, 
see (ii) of claim 1. 

(4) Let now tt = /3 o a be the morphism ly (G, A) — > Z, given by x t-^ ((a), (-21^2 — -22^1)), 
where zi{a) = Z2{a) = 0. It is obviously G-equivariant and its fibres coincide with the 
G-orbits. Since a is a geometric quotient and /5 is an affine bundle, then n is also a 
geometric quotient. 
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Remark: The variety Z is isomorphic to the moduh space M = Mp2(3m + 1) of stable 
coherent sheaves on P2 with Hilbert polynomial x^{f^) = 3m + 1. This had been verified 
by J. Le Potier in [^|. The space W^{G,A) is a natural parametrization of M because 



any !F G M can be presented in an extension sequence Oc J-" ^ Cp ^ where C 
is the cubic curve supporting J-" and p E C, and then JF has a resolution 

^ 2C(-2) ^O(-l) © ^ ^ 0. 

This resolution is the Beilinson resolution as can easily be verified. Moreover, x is {G, A)- 
stable if and only if JF is stable. (If p is a smooth point of C, then JF is the line bundle 
Oc{p) and if p is a singular point of C, then JF is the unique Cohen-Macaulay module on 
C with the given polynomial). There is an obvious universal family J-' on W^{G, A) x 
which defines a G-equivariant morphism W^{G,A) M and then a bijective morphism 
Z M, which by smoothness, is an isomorphism. One knows that M carries a universal 
family S. This family can be obtained as the non-trivial extension 

where H = FS^V* and Z XjjZ C Z X PV, or can be obtained as the descent of the family 



JF. More details can be found in [IS . 



4.2.2. The case yUi < 1/2 

We suppose now that the polarization A is such that fii < 1/2. In this case an element x 
of W is G-stable if and only if zi, Z2 are not both zero, and if for every z E V* , qi — zz\ 
and g2 — ZZ2 are linearly independent. 

4.2.3. Proposition: For this polarization there does not exist a good quotient 
W'{G,A)//G. 

Proof. Let Zi be a non-zero element of V*, let q G S'^V*\ziV* , and let x G be the 
matrix 

' zi 



q z\ 



Then x is stable. 



Claim : The orbit Gx is closed and if y E W^{G,A) is such that Gy meets Gx, then 
y G Gx. 



Before proving the claim, we will show that it implies proposition ^.2.3| . The stabilizer of 
a generic point in W'^{G, A) is isomorphic to C* : it consists of pairs of homotheties (A, A). 
It follows that if M = W^IG, A)/ /G exists, then all the fibers of the quotient morphism 
71 : W^{G, A) — > M are of dimension at least dim(G') — 1. Now suppose that the claim is 
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true. Then this imphes that vr ^(7r(x)) = Gx. But the stabihzer G^ of x has dimension 
2 : it consists of pairs 

■ a \ / a 

(3 a J \ I3zi a 

with a G C*, /9 G C, and hence has dimension 2. It follows that 
dim(7r^^(7r(x))) < dim(G') — 1, a contradiction. 

Proof of the claim. Let y G W^iG, A) such that x G G?/- Let 



y 



Z Z2 



Then zi is contained in the vector space spanned by z and Z2- Hence by replacing y with 
an element of Gy we can assume that z = Zi and that ^2 = if Z2 is a multiple of Zi. 



According to lemma [4.2.4| there exists a smooth irreducible curve C, xq G C, and a 
morphism 

: C\{xo} — > G 



such that 



W 



can be extended to 9 : C - 

m 



e : C\{xo} - 
1 1 — 

W, with ^^(xo) = x. We can write, for t G C\{a;o}, 



Oit)y 



a{t)zi + h{t)z2 c{t)zi + (i(t)2;2 



with 



(1) 
(2) 



gi(t) = X{t){a{t)qi + h{t)q2 + u{t)zi), 

q2{t) = X{t){c{t)qi + d{t)q2 + u{t)z2), 

where X, a, b, c, d are morphisms C\{xo} — > C and u : C\{xo} — > V*. The morphisms 
A, a, b, c, d can be extended to morphisms C — i>Pi = CU{0,oo}, denoted by A, a, b, 
c, d respectively, and u extends to u : C ^ F(y* © C). Now we use the fact that 6 is 
defined at Xq. The first consequence is that a{xo) = 1, c(xo) = 0, and if ^2 7^ then b 
and d also vanish at Xq. The second is that the morphisms qi,q2 '■ C\{xo} S'^V* can 
be extended to qi,q2 '■ C ^ S'^V*, and we have qi{xo) = q, q2{xo) = zf. 

We will now consider three cases : A(a;o) = 0, A(a;o) = oo, X{xq) G C*. 

Suppose that A(a;o) = 0. If Z2 7^ 0, then (1) implies that qi{xo) = g is a multiple of zi, 
but this is not true. If ^2 = then (2) implies that q2 is a multiple of zf and (1) implies 
then that q is also a multiple of zi, which is not true. Hence we cannot have A(xo) = 0. 
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Suppose that X{xo) — oo. If Z2 ^ 0, then (1) imphes that 

fi : C\{xo} ^ S^V* 

1 1 ^ a{t)qi + b{t)q2 + u{t)zi 

and 

77 : C\{xo} ^ S^V* 

1 1 ^ c{t)qi + d{t)q2 + u{t)zi 

extend to morphisms C S'^V* which vanish at xq. It foUows from the fact that fi{xQ) = 
that u = u{xo) G V*, and that qi = —uzi. Since qi 7^ (by G-stabihty of y), we have 
u 0. But since c{xq) = d{xo) = 0, this contradicts the fact that r]{xo) = 0. 

If 2:2 = then we deduce from the fact that n{xo) = that qi G <q2,V*zi>, which 
contradicts the G-stabihty of y. 

It follows that we have S = A(xo) G C*. If Z2 7^ 0, using the fact that a{xo) = 1 and 
6(,To) = c{xq) = d{xQ) = we see that u — u{xo) G V* and that zf — Suz2, which con- 
tradicts the fact that zi A Z2 ^ 0. 

Hence we have Z2 — 0. It follows from (2) that d{xo) G C* and that zf — Sd{xo)q2- By 
(1) we see that 

e : C\{xo} S^V* 



extends to C and that 



We have, if t ^ Xq 



1 1 > b{t)q2 + u{t)zi 



dd[Xo) 



It follows that e{xo) is a multiple of zi : e{xo) = ziv. We have then 



and 



as claimed. 



qi = -^q-ziv 



11 12 J \si- zi" ai^''' 



It remains to show that Gx is closed. This can be proved easily by computing the 
stabihzers of all the points in W^{G, A). We see then that G^ has the maximal possible 
dimension, hence Gx is closed. □ 
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We now give a proof of the lemma used in the preceeding proposition : 

4.2.4. Lemma : Let W be a finite dimensional vector space, G a linear algebraic group 
acting algebraically on W , y &W and x e Gy\Gy. Then there exists a smooth curve G, 
Xq & G and a morphism 

e : G\{xo} G 

such that the morphism 

: G\{xo} W 

1 1 e{t)y 

extends to 9 : G ^ W and that 9{xq) — x. 

Proof. Let n = dim(W^), d = dim{Gy). The generic {n — d + l)-dimensional affine 
subspace F G W through x meets Gy on a curve, and meets Gy\Gy in a finite number 
of points. Hence we can find a curve X C Gy that meets Gy\Gy only at x. Taking the 
normalization of X and substracting a finite number of points or unnecessary components 
if needed, we obtain a morphism a : Z — > Gy (where Z is a smooth curve) and a point 
zq & Z such that a{zo) — x and a{Z\{zQ}) C Gy. Consider now the restriction of a 

Z\{zo} ^Gy^ G/Gy. 

There exists a smooth curve Z' and an ctale surjective morphism : Z' — Z\{zq} such 
that the principal Gj^-bundle (f)*a*G on Z' is locally trivial. By considering completions 
Z', Z of Z', Z and an extension of to a morphism Z' ^ Z we obtain a smooth curve 
Y, yQ G Y and a morphism P :Y ^ Z such that P{yo) = Zq and that the principal Gy- 
bundle F = P*a*G is defined on Y\{yo} and locally trivial. Let U C F be a nonempty 
open subset such that we have a G^-isomorphism 

J : T\u c:^ U X Gy 

Then we can take G — U Li {yo}, Xq — yo, and for t e G\{xo} = U, we have 

e{t) = ^(r\t,e)), 

where ijj is the canonical morphism F — > G. □ 



4.3. More general counterexamples of inexistence of geometric quotients 
Let W be the space of homomorphisms 

o{-2) e o{-i) c^" o(i) 
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over P„ and let the homomorphism (f)Q G W he given by the matrix 



Zozj 







V zozl / 

where the Zy are homogeneous coordinates. The stabihzer of (po contains C* and the pairs 



1 

azo 1 

->2n\ 







-a/„ 



in Aut(C(-2) © C(-l)) x GL(C2") and thus has dimension > 2. If A = (Ai, Aa, -/xi) is 
a polarization with < Ai, < A2 < ^, then it is easy to see that 0o is A-stable in the 
sense of |4.1| . For example (m'^, m'g, n') = (0, 1, n) is the dimension vector of a (/)o-invariant 
choice of subspaces with Aim'^ + A2^2 ~ f^i^' = A2 — 1/2 < 0. There are however stable 
homomorphisms (p & W with stabilizer C*. Therefore W^{G, A)/G can never admit the 



structure of a geometric quotient. We will see in 7.2.2 that a sufficient condition for that 



in the case of this W is X2 > {n + l)Ai or A2 > because Ai + A2 = 1. 



5. Embedding into a reductive group action 

We will construct an algebraic reductive group G, a finite dimensional vector space W 
on which G acts algebraically, and an injective morphism 

C:W — ^ W 

compatible with a morphism of groups 

e-.G — ^ G. 

The traces of G-orbits on C(^) be exactly the G-orbits. The space W is of the 
same type as those studied in |3Tl| . We will associate naturally to any polarization of the 
action of G on W a character x of G/C*, i.e. a polarization of the action of G on W. 
We will prove that in certain cases a point w of is G— (semi-) stable with respect to 
the given polarization if and only if ({w) is x-(semi-)stable with respect to the associated 
polarization. The existence of a good and projective quotient of the open set of G-semi- 
stable points will follow from this. 
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5.1. Motivation in terms of sheaves 

The idea for the embedding of W into a space W with a reductive group action is to 
replace the sheaves Si in £ — (g) Mi) by £i (8) Hom(£^i, £j) and dually the sheaves J^i 
in = (B{J^i (8) Ni) by JF, (g) Hom(jF;, JF^)* and then to consider the induced composed 
homomorphisms 7($) for $ e Hom(£,^) = 

£i (g) Hom(£i, 8) ^ 8 ^ T ^ Ts® Hom(JP, JFJ* 

in the bigger space W of all homomorphisms between £i ® Hom(£^i,£^) and !Fs ® 
Hom(J^, ^5)*. This space is naturally acted on by the reductive group 

G = GL(Hom(£:i,£:)) x GL(Hom(:r, JF,)*). 

However it is not suitable enough for our purpose by two reasons. It does not allow enough 
polarizations as in section 3 for direct sums in order to have consistency of (semi-) stability 

and, secondly the group actions G^W and GxW ^ W don't have consistent orbits. 

Both insufficiencies are however eliminated when we consider the following enlargement 
of W. We set 

Pi = Hom(£^j,£^) and Qi = Hom(jF, JF;)*, 
and introduce the auxiliary spaces 

Wi= © Hom(Pi0Hom(£i_i,<E:i),Pi_i), W^^ = © Hom(Qi+i (g) Hom(JP,, JF^+i), Q,), 

l<i<r l<l<s 

and define 

W = Wl © Hom(£:i ® Pi, JF, ® Q,) © Wr. 
There are distinguished elements 

,^r) e Wi, {%,■■■ ,Vs^i) eWn 

whose components are the natural composition maps. The embedding of W into W will 
be defined as the affine map 

w^w, $ ^ ((6, ■ ■ ■ , ^r),im, im, ■ ■ ■ , vs-i)), 

where 7($) is the above composition for a given ^ E W. The components of and 
W/? will guarantee a compatible action of a reductive group and at the same time the 
possibility of choosing enough polarizations for this action. 

5.1.1. Remark; One might hope to be able to do induction on r and/or s by simply 
replacing M^-i (g) 8r-i © Mj. (g 8r by {Mj—i © Mj. (g Hom(£^r-i, ^r)) ® ^r-i and keeping 
the other for i < r — 1. But then we drop the information about the homomorphisms 
8i — > 8r. Therefore we are lead to replace all 8i, i > 2, by 81 at a time, i.e. by 

Pi (g £1 = (Ml © M2 (g A21 © • • • © Mr (g Ari) (g 81, 
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where Aji = B.om{Si,Sj). Moreover, in order to keep the information of the homomor- 
phisms Si — > Sj for 2 < i < j we consider also the spaces 

Pi = Mi® Mi+i ® Ai+ij ®...®Mr®Ari 

together with the maps Pi ® —>■ Pi-i in the following. The reader may convince 

himself that only because of this the actions of the original group is compatible with the 
action of the bigger reductive group. It is a beautiful outcome that then we are able to 
compare the semi-stability with respect to related polarizations in section 7. 



5.2. The abstract definition of W 

The above motivating definition of the space W can immediately be turned into the 
following final definition using the spaces Hu, Aji and B^i and the pairings between them. 
For any possible i and / we introduce the spaces 

Pi= ® Mj ® Aji and Qi = ® Nm® Bi^, 

l<m<l 

and we denote by pi and qi their dimensions. For 1 < i and Z < s we let 

Pi Ai^i_i Pi^i and Qi^i ® Bi+i^i Qi 

be the canonical morphisms, defined as follows. On the component Mj Cg) Aji of P,, the 
map ^i is the map 

(Mj ® Aji) ® Ai^i^i — > Mj ® Aj- i_i 

induced by the composition map of the spaces A. The map t]i is defined in the same way. 
As in we set 

Wl= ® Rom{Pi0Ai^i_uPi-i), Wr= ® Rom{Qi+i0Bi+i^i,Qi), 

l<i<r l<l<s 

and 

W = Wl® Hom(Pi, Qs ® H,i) © W^. 

In order to define the embedding ( we define the operator 7 as follows. Given w = {(pu) G 
W with (pii e Hom(Mi, A^; Hu), we let 

-f{w) e Hom(Pi, Qs (g) Hsi) = Hom(Pi ® H*^, Q,) 

be the linear map defined by the matrix {•ju^w)), for which each •Jh^w) is the composed 
linear map 

Mi ® Aa — ^Ni(g Hu ® Aa — > Ni (g Hn — ^ iV^ ® B^^ ® 

where the first map is induced by (pu, the second by the composition Hu An Hn and 
the third by the dual composition Hn B*i (g) Hsi. 
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The map ( can now be defined by 



W^W, ((6, ■ ■ ■ , ^r),l{w), (r^i, ■ ■ ■ , r/,_i)). 



5.2.1. Lemma: The linear map 7 is injective and hence the morphism ( is a closed 
embedding of affine schemes. 

Proof. From the surjectivity assumptions in |2.1| we find that dually the composition 

is injective. Now it follows from the definition of 'jii{w) that (pa can be recovered from 
•yii^w), by shifting An to its dual. □ 



5.3. The new group G 



We consider now the natural action on W as described in in the general situation, 
where the group is 

G = GlX Gr, with Gl= H GL(PO, G« = n ^UQi)- 



l<i<r 



KKs 



To be precise, this action is described in components by 

gi-i o Xi^i^i o ^ idy^ , hso^ o {g^®idY^ and h o y^^^^ o {hi+i ® id)'^ , 

with 

Xi-i^i G Hom(Pi(g)Ai,i_i,Pi_i), iIj E Rom{Pi®H*^,Qs), yi,i+i E Rom{Qi+i(g)Bi+i^i,Qi) 
and with 

g., E GL(P,), hi E GL{Qi). 
The first and third expression describe the natural actions of on and of Gr on 

There are also natural embeddings of Gl, Gr, G into Gl, Gr, G respectively. For that 
it is enough to describe the embedding of Gl in G/,. Given an element g E Gl, 

( gi ... \ 

U21 g2 '■ 

9 = 

; ■•. ■•. 

\Url ... Mr,r-1 Or j 
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with gi e GL(Mj) and Uji e Hom(Mj, Mj <S) Aji) we define 0^i{g) e GL(Pj) as the matrix 

/ ... \ 

1*1-1-1,1 ^i-l-l • 

; ■•• ■•• 

y '^r,i ■ ■ ■ ^r,r— 1 9r j 



with respect to the decomposition of Pi with the following components: Qj — gj (8) id on 
Mj (g) Aji and for i < j < k the map itfej is the composition 



Mj (g) Aji 



Mk ® Akj ® Aji — >Mk® Aki, 



where the second arrow is induced by the given pairing. In case j = i we have gi = gi and 
Uki = Uki- Now we define the map 

Gl ^ Gl by g^ {OL,ig, ■■■ , O^vQ)- 



It is then easy to verify that 9l is an injective group homomorphism and defines a closed 
embedding of algebraic groups. With this embedding we consider G^, as a closed subgroup 
of Gl. In the same way we obtain a closed embedding Or of Gr C Gr. Finally we obtain 
the closed embedding 9 — [9^, 9r) of G C G. 



5.3.1. Lemma: The subgroup Gl C Gl (respectively Gr C Gr) is the stabilizer of the 
distinguished element (^2, • • • ,Cr) G Wl (respectively {rji, . . . ,ris-i) G 'Wr) 

Proof. It is enough to prove the statement only for G l because of duahty. The fact that 
Gl stabilizes (^2, ■ ■ ■ ,Cr) is an easy consequence of the properties of the composition 
maps. The converse can be proved by induction on r. It is trivial for r = 1. Suppose 
that r > 2 and that the statement is true for r — 1. Let (71, . . . ,7^) be an element of 
the stabilizer of • • • ,Cr)- When we replace the space W by W, corresponding to the 
spaces M2, . . . , Mr and the same spaces Ni and similarly Wl by W'l, then (72, . . . , 7r) 
is an element of the stabilizer of (^3, ... , ^j.), so by the induction hypothesis it belongs to 
and there exists an element 

(92 ••• \ 

^ «32 93 '■ 
': ■•. ■•. 

y Ur2 ■ ■ ■ Ur, r-l 9r J 

such that (72, ... , 7r) = Let now 71 e GL(Pi) have the components 

Mi (g) Ail Mj (g) Aji for all l<ij <r. 



30 



J.M.DREZET AND G.TRAUTMANN 



The identity 7i o ^2 = ^2 ° 72 then shows that yji = for j < i, yu = gi ioi 2 < i and 
Vji ~ '^ji 2 < j < z. Now let gi = yu, Uji = yji, for 2 < j < r, which are hnear 
mappings Mi — > Mj ® Aji. Then 

f 91 ■■■ \ 



9 



U21 92 



: ■•• ■•• 

y Url ■ ■ ■ Ur,r-1 9r ) 

is an element of Gj^ and we have (71, . . . , 7r) = Qhig)- 



□ 



Remark: since the action of on W^^ is linear, it is clear that we have an isomorphism 

Gl/Gl ^ Gl(^2, • • • ,^r), and similarly Gr/Gr ^ Gr{t]i, . . . ,r]s^i). 
We will use this fact in section 8. 

Using the associativity of the composition maps it is again easy to verify that the actions 
of G on and G on W are compatible, i.e. that the diagram 

GxW > W 



GxW 



W 



is commutative, in which the horizontal maps are the actions. In addition we have the 



5.3.2. Corollary: Let w,w' G W . Then w and w' are in the same G-orbit in W if and 
only if ({w) and C(w') are in the same G-orbit in W. 

Proof. It follows from the compatibility of the actions that if g.w = w' in W then also 
9{g).({w) = C{w') in W by the last diagram. Conversely, if g G G and g-C{w) = ({w') 
then g stabihzes (,^2, ■ ■ ■ ,C,r,Vi^''' ; Vs-i) by the definition of ( in |^ . By Lemma |5.3.1 
geG. □ 



5.4. The associated polarization 

In p.3.1| and p.3.2| we had introduced polarizations for the different types of actions of 
Gred on W and of G on W. In the following we will describe polarizations on W and W 
which are compatible with the morphism C, : W — > W. Their weight vectors are related 
by the following matrix equations and determine each other. The entries of the matrices 
are just the dimensions of the spaces Aji and Bmi- 
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In the sequel we will use the following notation: the dimension of a vector space will be 
the small version of its name. So = dim(Mj), rii = dim{Ni), pi = dim(Pi), = 
dim(Qm) aji = dim{Aji), bmi = dim{Bmi) etc. 

A proper polarization of the action of G on is a tuple A = (Ai, ... , A^, — /ii, . . . , — /^s), 
where Aj and fii are positive rational numbers such that 

^ Xiirii = ^ niui = 1. 



l<j<r 



KKs 



We define the new sequence of rational numbers ai, . . . ,ar, Pi, ■ ■ ■ , Ps by the conditions 



f \i\ ( I ••• Q \ ( ai\ 

(121 1 '■• : 

; ■•. ■•. 

y A,. I \arl ■■■ ar,r_l 1 y J 



f / 1 b2,i ■■■ bsi \ f Pi\ 

1 ■•. ! 



v 







1 



Then we have 



1 = Airrij = ttiPi and 1 = fJ'ini = PiQi- 



>i " "i 

l<j<r l<i<r 



KKs 



KKs 



In particular the tuple A = ■ ■ ■ , a^, —Pi, ■ ■ ■ , —Ps) is a polarization on W such that 
ai is the weight of Pi and —Pi the weight of Qi. It is called the associated polarization on 
W. It is compatible with C, in the following sense: If M[ C Mi, and N[ C A*"; are linear 
subspaces, and if the subspaces of Pi and Qi are defined by 

P[= ® M' ® Aj„ and Q'l = ® ® B*^i 

i<j l<m 

respectively then we have 



l<«<r 



l<j<r 



mi 

KKs KKs 



If the set of stable points in W with respect to the associated polarization is non-empty 
then by |3.3.2| the weights satisfy the conditions 

ajPj > for any i and Piqi > for any m. 

Kj<r 



KKm 



Equivalently the conditions may also be written as 

ajPj > for 2 < i < r and 1 — Piqi > for 2 < m < s. 

i<j<r 



m<Ks 



Substituting the weights of the original polarization on W , we can reformulate these 
conditions. In the cases treated in the examples they reduce to the following 
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5.4.1. Weight conditions. 

Let W be of type (r, s) and let A = (Ai, ... , A^, — /ii, • • • , — /^s) be a proper polarization 
of W with positive Aj and fii. If the set W*(G, A) of stable points of W with respect to 
the associated polarization A is non-empty, then in case of 

type (2, 1): Aa - aaiAi > 0, 

type (3,1): A3 - OsaAa + (032021 - 031)^1 > 0, Ai(mi + a2im2 + asims) < 1, 
type (2,2): A2 - aaiAi > 0, /ii - ^21/^2 > 0. 



5.5. Comparison of invariant polynomials 

In the following we assume that A = {ai, . . . , a^, — . . . , —f3s) is the polarization on 
W associated to the polarization A = (Ai, ... , A^, — /^i, . . . , —/is)- The semi-stable locus 
W**(G, A) with respect to this polarization is more precisely defined by the character A" 
associated to it as in pTT] . If q is lowest common denominator of ai, . . . , a^, . . . , /^s, we 
have 

X{g) = ( n detigr'"')i n detih^yf^^) 

l<i<r l<l<s 

for an element g e G with components gj and h;. By the matrix relations between the 
polarizations q is also a common denominator of Ai, . . . , Ar,yUi, ... ,f^s, such that, if p 
denotes the lowest, we have q = pu for some u. The character x with respect to the given 
polarization can be defined by 



X{g,h)= H detig.r^^^ H det{h 



l<i<r KKs 



where the gi resp. hi are the diagonal components of g resp. h, see p.2| . Now the relations 
between the polarizations imply by a straightforward calculation that 

X{9{g,h)) = x{g,hY. 

If F is a A''"-invariant polynomial on W it follows that 

F(C(((7, h).w)) = F{e{g, h).aw)) = x{9, /i)"™F(C(u;)), 

i.e. that F o ( is a x^^-invariant polynomial on W. As a consequence we obtain the 



5.5.1. Lemma: C"HW^'(G,A)) C W''{G,A), 

i.e. if w & W and C{w) is G- semi- stable in W with respect to the polarization A = 
(ai, . . . , ttr, —Pi, • . • , —Ps) then w is G -semi-stable in W with respect to the polarization 
A = (Ai, ... , Ar, — /ii, . . . , — yUs) (in the sense of \4.1\ ). 
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Proof. There exists a ^''"-invariant polynomial F on W such that F{({w)) ^ 0. Then 

F{a{g,h).w)) = F{C{w))^0 

for any element {g, h) in the unipotent subgroup H (Z G. This means that w is G-semi- 
stable. □ 

5.5.2. Remark: When we consider the subgroup G' C G defined by the condition 

det(gi) = det(h,) = 1, 



we have 6{G') C G' as follows from the definition of G' in |3.5| . With respect to these 
groups the semi-stable points are those over the semi-stable loci in P(W^) resp. P(W), 
with respect to the line bundles 

£ = Op(iy)(t) and L = Op(w)(t), 



where t and t is defined as in |3.4| in the different cases endowed with the modified action 
defined by the characters. However, we cannot compare P(iy) and P(W) directly because 
the morphism C, does not descend. 



We need the analogous statement of Lemma p.5.1| also in the case of stable points. For 



that is is more convenient to use the subspace criterion (1) of A. King in the case of Gred 
and G. This gives also another proof in the semi-stable case. 



5.5.3. Lemma: With the same notation as in the previous Lemma 

C^^(W^(G,A)) C W'{G,A) 

Proof. Let w = {(pu) be a point of W with maps Mj H^- A*"; and suppose that w is 
not G-stable with respect to the polarization A. We can assume that it is not Gred^stable, 
too. Then there are linear subspaces M- C Mj and A"/ C A^; for all i and / such that the 
family ((Mj)), (A"/)) is proper and such that 

MM:^H;^cNI and A^m^ - > 0. 

i I 

With these subspaces we can introduce the subspaces P.- C Pi and C Qi as 

P-=® Mj ® Aji and Q'l = ® N^® B*^^. 

i<j m<l 

They form a proper family of subspaces and satisfy 

for the possible values of i and /. But by the definition of the spaces and because A is the 
associated polarization, the formulas of |5.4| imply the dimension formula 

^ aip[ - ^ Piq'i = Xim'i - ^ fun'i > 0. 

i i I 
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This states that also C{w) is not G-stable. □ 

In section 7 we will derive sufficient conditions for the equality 

C"^(W'^(G, A)) = W'{G, A) and C"^(W^'(G, A)) = W''{G, A). 

In the following section we show how this equality implies the existence of a good and 
projective quotient W'^'^{G, A)/ /G using the result for W*'*(G,A)//G from Geometric 
Invariant Theory. 



6. Construction and properties of the quotient 

We keep the notation of the previous sections and let A be the polarization on W asso- 
ciated to the polarization A on W. We do not require that they are proper here, but we 
will do that later for the examples. In addition we introduce the saturation 

Z = GC{W) C W 

of the image of W with respect to the action of G. 

6.1. Construction of the quotient 

6.1.1. Proposition.- Let W and W together with their G- and G-structure be as in 
section 2 and 5, let A he a polarization for (W^ G) and A be the associated polarization 
for (W,G). 

(1) If C""^(W*(G, A)) = W%G, A), then there exists a geometric quotient W%G, A) — ^ 
M'^ of by G, which is a quasi-projective nonsingular variety. 

(2) If m addition C~^(W^'(G, A)) = W{G,A) and {Z \ Z) n W{G,A) = 0, 
then there exists a good quotient W^^{G,A) M, such that M is a normal projective 
variety, is an open subset of M, and VI^*((j', A) is the restriction of n. 



We recall here the definition of a good and a geometric quotient of C.S. Seshadri, see p9| , 
|27|. Let an algebraic group G act on an algebraic variety or algebraic scheme X. Then 



a pair (y), Y) of a variety and a morphism X — ^ F is called a good quotient if 

(i) ip is G-equivariant (for the trivial action of G onY), 

(ii) ip is affine and surjective, 

(iii) If U is an open subset of Y then yj* is an isomorphism Oy{U) ~ where 
the latter denotes the ring of G-invariant functions, 
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(iv) If Fi,F2 are disjoint closed and G-invariant subvarieties of X then ip{Fi) , ip{F2) are 
closed and disjoint. 

If in addition the fibres of (p are the orbits of the action and all have the same dimension, 
the quotient {ip, Y) is called a geometric quotient. 

As usual we write X/ /G for a good quotient space and X/ G for a geometric quotient 
space. 

Proof. We will prove the second statement first, assuming that the conditions of (1) and 
(2) are satisfied. We use the abbreviations W = W''{G, A),W^^ = W"^(G,A) and 
similarly W^, for the subsets of the stable points. By the result of A. King, there 
exists a good projective quotient of W*'' by the reductive group G. So there exists also a 
good and projective quotient of the closed invariant subvariety Z fl W** which we denote 
by 

z n w ^ M. 



By assumption (2) G({W'') = Zn W = Z n W. We let vr be the composition 



■GCiW 



SS\ 'TO 



M. 



We know already that M is projective. We will then verify that (vr, M) is the good 
quotient of the proposition. We consider first the commutative diagram 



p 



M 



in which p is the projection and /i is defined by (g, w) ^— gC(w). There is an action of G 
on G X by g.{g„w) = {g9{g)~^, g.w) and it follows that /i is G-equivariant. 



Claim: The morphism fi is a geometric quotient of G x W^'^ by G. 

Proof of the claim: We show first that the fibres of /i are the G-orbits. So let 
{g,w) , {g',w') be two elements in G x W'^'^ such that /i(g, w) = fi{g',w'). Then 
({w) = g~"^g'C('U^')- By Lemma p.3.1| g = g'^g' G G and g.{g,w) = {g',w'). The 
claim will be proved if we show that fi has local sections. For this it suffices to use the 
remark following Lemma 5.3.1 and a local section of the quotient map G — G/G. 

Now we are going to verify the 4 properties of a good quotient for vr. Clearly (i) is satisfied 
by the definition of vr. 

Proof of (ii). It is clear that tt is surjective. The morphism n is affine because tt = ttq o C 
and ttq and ( are affine. 
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Proof of (iii). Let U G M he an open subset. Then 

0{U) C 0{7r-\U)f 

since vr is G-invariant. Conversely let / G 0{'k^^{U))^. The f op e 0{G x 71^^(11))^, 
and since /i is a geometric quotient, fop descends to an / G C(/i(G x n^^{U))), which is 
G-invariant. Now again / descends because ttq is a good quotient. This proves equahty 
0{U) = Oin-\U)f. 

Proof of (iv). Let Fi,F2 be disjoint, closed, G-invariant subvarieties of W^'^. Then 
p~^ (Fi) , p^^ {F2) are disjoint, closed and G-invariant subvarieties of G x W'^'^. Since /i is 
a good quotient, /i(j9~^(F2)) are disjoint, closed and G-invariant in G((W^'^). 

Finally, since ttq is a good quotient, ttq o/i(p~i(Fi)), ttq o/i(p~i(F2)) are disjoint and closed 
subvarieties of M. But ttq o jj,{p~^[Fi)) = ir^Fi), which proves (iv). 

The normality of M follows from the fact that GCllV^^) is smooth and ttq is a good 
quotient, [2^, with respect to the reductive group G. That vr becomes a geometric 



quotient on the open set of stable points follows from the fact that the G-orbits in 
G({W'') = Z n W'^ intersect in G-orbits. In particular the stabilizers of in G and 
of C(^) ill G are isomorphic, such that all orbits have the same dimension. 

The proof of (1) is a modification of the above. In any case ttq induces the geometric 
quotient Z ^ Mq with Mq open in M. Now GC{W') = Z n WMs a vro-saturated 

open subset of Z n W, such that we obtain a geometric quotient GC(W^) with 
C Mq open. By the same arguments as above applied to the diagram related to 
G X — >• GCCW^) we conclude that is a geometric quotient. □ 



Remarks: 1) The idea of this proof comes from 0|, and has already been used in [|T3 
and §. 

2) If the second condition of (2) is not satisfied, we cannot even prove that W*(G, A) 
admits a good quasi-projective quotient, because Z D might not be saturated. Of 
course the projectivity of the quotient depends on this condition. 



6.2. S-EQUIVALENCE 

We suppose that the hypotheses of proposition |6.L1| are satisfied, with polarization A for 
{W, G) and associated polarization A for (W, G). 

It is easy to define the Jordan-Holder filtration of G-semi-stable elements of W with 
respect to A (cf. |^ for a more general situation). Using the preceding results we can 
also define a Jordan-Holder filtration of a G-semi-stable element of W with respect to A. 
Let w = {(pii) G W^'^{G, A). Then there exist a positive integer p, an element h E H and 
filtrations 

M° = {0} C M/ C • ■ ■ C Mf = Mi, = {0} C A^/ C ■ ■ ■ C Nf = Ni, 



QUOTIENTS BY NON-REDUCTIVE GROUPS 



37 



with 

J2 dim(M/) = J2f^i dim(iV/) 

i I 

for each j, such that h.w = {(pu) satisfies 
and that if 

is the induced morphism, then {4>ji)ii is G-stable with respect to A for any j. This 
filtration and h need not be unique, but p is unique and the too, up to the order 

and isomorphisms. Conversely, an element of W having such a filtration is G-semi-stable 
with respect to A. We say that two elements {(pu) and of VI^*''(G, A) are S-equivalent 
if they have Jordan-Holder decompositions {4>li), {(p'u) respectively of the same length, 
and if there exists a permutation a of {1, . . . ,p} such that {(p'li) is isomorphic to {(f)'^}''^) 
for any j. 

The following result is also easily deduced from |6.1.1| . 

6.2.1. Proposition; Letw,w' E W^''{G,A). Themi{w) = n{w') if and only if w and w' 
are S-equivalent. 

It follows that the set of closed points of M is exactly the set of S-equivalence classes of 
elements of W^^. 



7. Comparison of semi-stability 

We are going to investigate conditions for the weights of the polarizations under which a 
(semi-)stable point w E W is mapped to a (semi-)stable point Cl"^) ^ W. For the 
estimates we need the following constants which depend on the dimensions rrii and the 
composition maps Hu An — >• Hn. 

7.1. Constants 

Let /C be the family of proper linear subspaces 

K C © Mi ^ Ail 

2<i 
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such that K is not contained in © M[®Aii for any family (M') 7^ {Mi) of subspaces. For 
any / we let the map 

2<i 2<i 

be induced by the maps An © H^^ — > H^- associated to the composition maps, which are 
supposed to be surjective, see We introduce the constant 

Q 7712, ... , rrir) = sup p^^ with pi{K) = . 

K&K codim(A) 

Similarly we define the constants di{ni, . . . ,ns-i) in the dual situation. Let 

© N; © if* ^ © iV; © Bsi © 

1<S 1<S 

be induced by the maps B^i © H*- if;* and let C be the family of proper subspaces 

Lc ®n;® Bsi 

1<S 

which are not contained in © © Bsi for any family (Nl) 7^ (N*) of subspaces. Then we 

Ks 

define 

, , codim((5,^(i:©ii*)) 

di(77) = di(77i, . . . , Us-l) = sup — — . 

Le£ codim(L) 



7.1.1. Lemma: If mi < mt for all i > 2, then Ci{m2, ■ ■ ■ ,mr) < Ci{m2, ■ ■ ■ ,mr) . 

Proof. It will be sufficient to assume that 777 j = 771 j for all i except one, 7772 < 7772 say. 
Then let Mi be vector spaces of dimensions 777^ and suppose that 

M2 = L2® M2 and Mi = Mi for i > 3. 

For any K E IC we consider the subspace 

i? = (L2 © A21) © i^ C (M2©A2i)©(©M,-© A-i). 

2<i 

Then codim(^) = codim(ir) and also codim(5; © iifj) = codim(5i (iC © iif^)) because 
61 is a direct sum of the surjective operator Aji © H^^ — > ii^* such that 6i{L2 © A21 © ii^* ) 
equals L2 © ii;*2 and Si{K © ii;* ) = (L2 © iif2) ® 6i{K ii;* ). Therefore pi{K) = pi{K). 
Once we have shown that also K belongs to the analogous family IC, the Lemma is proved. 
To see this let C M2 and M- = M^ C Mj for i > 3 be subspaces such that 

K C ©M;© Ail. 

2<i 

Then in particular 

i^2 ® A21 C M2 ® A21 
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and thus L2 C M^. But then M^ = L2® with = M2 n M2 and it follows that 

K c © m; ® Ail. 

2<i 

Since K & K, we obtain M- = Mi for all i and then also = M2. □ 



7.2. Study of the converse I 

Let A = (Ai, . . . Ar-, — /ii, . . . , —fJ's) be a polarization on TV and let A = (ai, ... , a^, 
— , — /?s) be the associated polarization on W (the associated polarization has been 

defined in |5^ ). We had shown in p.5.1| and |5.5.3| that if w E W and C{f^) is (semi-)stable 
in W with respect to G and A, then so is w with respect to G and A. We are going 
to derive sufficient conditions for the converse, i.e. whether C(^) is (semi-)stable if w is 
(semi-)stable. 

In the sequel we are going to use the following notation: Given a family M' = {M[) of 
subspaces M- C Mj we set 

Pi{M') = © M'^Aji 

and call a subspace P/ C Pi saturated if there is such a family with P- = Pi{M'). Note that 
in this case J^^^iP'i = S ■^I'^i- Similarly we introduce the spaces Qi{N') for a subfamily 

i i 

N' = (Nl) of (A^";) and call them saturated. 

Let w = {(pii) be given and assume that C{w) is not semi-stable with respect to A. Then 
there exist linear subspaces P/ C Pj and Q[ C Qi such that 

Up:®A,_i)cpU 7H(A'®^:i)cQ1, miQ'i+i®Bi+i,,)cQ[ 

and such that 

i I 

where as before the small characters denote the dimension of the spaces. If there were 
subspaces M[ C Mi and Nl C Ni with PI = Pi{M') and Q'l = Qi{N') as in ^XS) , then 
'j{w){P[ © H*{) C Q', would imply that (puiM- © iJfJ C A^/ and we would have 

and w would not be semi-stable. In the following we are going to construct families 
M", A^" of subspaces M/' C Mi and A"/' c A"; such that PI' = Pi{M") and Q'l = Qi{N") 
are as close to P/, Q'l as possible and such that there is a useful estimate for 
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Step 1: We can assume that P/ has a decomposition 



Pl = M[®Xi in Mi®{® Mj®Aji) 

i<j 



and such that Xr = 0. To derive this, we remark that for a subspace of a direct sum 
E®F of vector spaces there exists a hnear map E ^ F such that the isomorphism "J) of 
E ® F transforms S into 5" © S"', where S' is the projection of 5" in ii^ and S" = S f\ F. 



Using this and descending induction on i we can find an element h G Hi C Gl, see p.4| , 
such that the truncations 9L,i{h) G GL(Pj), see |5]^, map P/ onto a direct sum M| © Xi 
for any z. Since ^i(P/ © C P/_i we easily derive that 

© M- © Aji C Xi C ® Mj® Aji 

i<j i<j 

for all possible i. We put 

Pi = codim( © M' © Aji, X,) = codim(P,(M'), P-). 

i<j 

Note that pr = 0. 

Step 2: Let M", . . . M^' be subspaces of Mi, . . . , respectively such that 

PiiM") D i^' 

is minimal over P/ for any i. Then M^' C since these spaces are the first components 
of P^ C Pi{M") respectively and we have M[ = M'{. We let 

a^ = - m'j)a,i = codim(P, (M'), Pi(M")). 

Step 3: We are going to define the subspaces N[ C A^"" C A^"; as images. 
Let Pi © H*^ Ni be the map which is the sum of the composed maps 

Mi © Ail © H*^ ^Mi® H*i ^ Ni. 

Then we define 

n; = ^i{w){p[ © H*,) = MM[ © h;,) + ^iiw){x, © h;,) 

and 

N[' = ^i{w){P,{M") © Ht,) = MM'I © HI,) + 5^ 0,,(m; © i/,;. )• 

2<i 

It follows N[ C A^/' for any /. 

Step 4- If the weights Pi are supposed to be positive, we may assume that 

7(w)(Pi' © h:,) = q; and r/z(Q;+i © p,+i,o = g; 
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for / < s. Otherwise we could choose subspaces Q'l C Q'l by descending induction as 

images. Then — ^ jSiql > — X] Pili would improve the assumption on the choice of the 

I 

spaces P- and Q'l. Now it follows that for any / 

Q'l c Qi{N") 

because P{ ® H*^ is mapped to © N[' (g) B*i and the maps rii are the identity on the spaces 

1<S 

N^. Note that we even have Q'l C Qi{N') since 7/ | P[®H*^ factorises through © N'^®B*i 

1<S 

as follows from the definition of N^. 



7.2.1. Lemma; Suppose that all /3i, ... , /3s > 0, and let A = Xim'- — ^ fiin'/. Then 

i I 

A > ^ A"?/ - X^/Ui^/ + ^aj((Ti - Pi) - ^fiici{m2, . . . ,mr)((Ti - pi). 
Proof. Let 

= 5,(^1 ® H*,) C = © ® 

Since Xi is not contained in a direct sum with spaces smaller than we get 

codim(r;, Zi) < ci{m'^, ... , m'^) codim(Xi, © M'' © Ai). 

2<i 

By Lemma |7.1.1| and above definitions we get 

codim(Y;,Zi) < ci{m'^, . . . ,m")(^m"aii - p\) = Q(m2, . . . ,mr)(^im'- - m-)aii - pi). 

l<i 2<i 

The map ^ sends (Mf © © Zi onto A^"/' by definition of Nj^' and also maps {M[ © 

i 

-^u) ® ^li^i ® Hi^) onto A^/. Therefore, since M[ = M(', we have a surjection 

^ iv/'/iv; 

and the dimension estimate 

n'l' -n'i< Q(m2, . . . ,mr)C^{m'- - m-)aii - pi). 

2<i 

Now we can derive the estimate of the Lemma. If there is no summation condition it is 
understood that the sum has to be taken over all indices of the given interval. We have 

i I 

= Yl - E /^i'^J + E - "^j) - E - n'l)- 

i I j I 

Substituting for A-,- in the third sum and replacing the first by 

Aim- = ai dim( © M' © Aji) = - pi) 

' ^ ' ^ i<j ' ^ 

i i i 
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and using the definition of cTj we get 

ill I 

Now using tlie assumed estimate for tfie first sum and tfie derived estimate for n!l — n[ we 

get 

A > ^ A?/ - + ^ai(cri - Pi) - ^pzQ(m2, . . . ,mr){ai - pi). 

I I i I 

□ 

7.2.2. Corollary; Suppose that s = 1, let A = (Ai, ... , A,., — ^) and let A be the associ- 
ated polarization (ai, . . . , a^, —:^)- If all > and if 

A2 > — Ci(m2, . . . ,mr) 
ni 

then 

C"^W*^(G, A) = W''{G, A) and C^W{G, A) = W'{G, A). 

Remarks: (1) Note that by the normalization of the polarizations we must have piUi = 1 
such that l/ni is the only possible value for pi = f3i. 

(2) If all ai > 0, then the necessary conditions for W^GjA) ^ and ^(GjA) 7^ are 



both satisfied, see |5.4| . The condition of the corollary is an extra condition. 



Proof. Let us first assume that C(^) is not semi-stable and let the spaces P[ and Q\ be 



as at the beginning of [7.2| . The only (5\ = 1/ni is positive. Let the other spaces be 
chosen as in |7^ . The difference Piq'i — J^f^i^'i reduces to q'l/ni — n[/ni, and since 
A'^^ = 'j{w){P[ ® -f^ii) = Q'l, this difference is zero. Therefore 

A > ai{ai - Pi) ci(m2, . . . ,mr){ai - pi). 

i 

Since all the are positive we have 

aiicTi - Pi) > ai(cri - pi) + 02(0-2 - P2). 

i 

Moreover, ^2 induces a surjection 

P2{M") ® A21/P2' ® Ai ^ Pi{M")/P[ 

because M[ = M". Therefore we obtain the dimensions estimate (o"2 — P2)a2i > o-i — pi- 
It follows that 

1 (X2 

A > ( Ci(m2, ... ,mr) + oi H )(o-i - pi). 

rii 021 
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Since A2 = a2iai + 012 > ^Ci(m2,... ,mj,) the last expression is non-negative. This 
proves the case of semi-stabihty. For the case of stabihty we assume that w is stable and 
that C('"^) is already semi-stable. If C('"^) were not stable, we would find subspaces P/ and 
as in such that ^ aip[ — fiin[ = and such that at least one P/ is different from 
Pi. Now let the spaces Mf and A''^ be constructed as above. Then we have 

A > > aiSi si > > ttiSi + (A2 021) — > 0, 

^-^ m ^-^ 111 a2i 

where Si = ai — pi = dim(Pj(M")/P/), and where we use that 52021 > Si. If the family 
M" is different from M, then > A, and if it is equal, then A = 0. In order to obtain a 
contradiction we have to show that M" is different from M. Assume that it is not. Then 
Si = dim(Pj/P/) and we must have Sj = for i > 3 and si(A2 — :^'^2i) = 0. If also si = 0, 
then by the above estimate also S2 = 0, contradicting the choice of the P/. Therefore 
Si 7^ and A2 = f[^CL2i- But then A = a2('S2 ~ ^) and we have 52021 = Si. From this it 
is easy to see that = Pi{M) where Mi = Mi ioi i ^ 2 and M2 = M^ ^ M2. Then we 
have 

aifhi - fiin[ = ^ \ip[ - fiin[ = 

i i 

which contradicts the stability of w. □ 



7.3. Study of the converse II 



We keep the notation of [7.2| and compare the (semi-) stability of points in W and W in 
two steps, each reducing to the case s = 1. We consider the intermediate space 

V = © © Hom(Pi © if* , Ni) 

1<1<S 

and the maps 

Here Ci is defined by 

W ^ (6, • • • ,^r,7l(w^), • • • ,7s(w)), 

where 7i(w) is the map defined hj w = {(pu) as in \l.2\ The map (2 is defined by 

(X2, . . . , x^, 7i, . . . , 7^ f-^ {X2, ... , Xr, 7, r^i, . . . , r/^-i), 

where now 7 : Pi © H*^ Qs is induced by the tuple (71, . . . ,7s) as the sum of the 
compositions 

Pi © Hl^ ^Ni® Ha © Hl^ ^Ni® B^i 
which are induced by the 7; and the pairings Bgi © Hn Hsi- It is obvious that 

C = C2oCi. 
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Note that both (i and (2 are injective by the same reason as for (. 
On V the group x Gr acts naturally and we have the embedding 

G = Gl X Gr '—>■ Gl X Gr, 

see f).'d\ . It follows as in section 5 that Ci is compatible with the group actions and 
that w,w' G W are on the same G-orbit if and only if (i{w),(i{'w') are on the same 
Gl X Gr orbit. Similarly we have the group embedding G^, x Gr ^ Gl x Gr = G 
and (2 is equivariant and satisfies the analogous statements for the orbits. Given the 
polarization A = (Ai, ... , A,., — /ii, . . . , —/is) for (W, G) we consider the polarization A = 
(«!, . . . , ttr, — yUi, • • • , —fJ's) for (V, Gl X Gr) where the ai are defined as in |5^ . As in 
[5.5. 1| , |5.5.3| it is easy to show that 

Cr'V^^(GL X Gr,A) C W{G,A) and Cr'V^(GL x Gr,A) C W'{G,A) 

and similarly that 

C2-^W^^(G,A) c Y''{Gl X Gr, A) and C^'W{G,A) c Y'{Gl x G^, A). 

Note that as for 14^**, W^, we have unipotent sub-orbits in V** and V^, see [4.1| . We are 
going to show that in all 4 cases equality holds under suitable conditions on the weights 
of the polarizations. Then the same is true for (. 



7.4. Estimate for Ci 

Let w = {(pii) in W be given and assume that Ci(^) is not semi-stable. Then there are 
linear subspaces P/ C Pi and C A*"; and a unipotent element h G Hr such that for 
(7i, • • • , 7s) = ^.(71, • • • , 7s) we have 

up: ® Ai^i_,) c and y^ip; ® h;,) c at; 

for alH > 2 and all /, and such that 

^aip- -"^fiin'i > 0. 

i I 

We may assume that h = id because Hr acts on W in the same way and we can replace 
w by h.w. Moreover, we may assume that all A"/ are equal to 'yi{P{^ H^^) since all fii > 0. 
Now we proceed as in replacing the spaces Qi by A";. Therefore we find subspaces 
C Ml' C Mi such that M[ = M'l and such that 

P[ = M[®Xi, P,{M') CP^C P,iM") 

and the family M" is minimal with this property. We denote 

A = codim(P,(M'), P^) , a, = codim(P,(M'), Pi{M")) 

and let 

N'/ = ^l{P^{M") H;,) D Ni 
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As in [7.2. 1| we consider the surjection 

Z^lYi ^ iV/'/iV/, 

where Yi C Zi are the same, and we get the estimate 

nl -n'i< Q(m2, . . . ,m,.)((Ti - pi) 
for any /. The estimation of the discriminant A is now simpler than in 17 



7.4.1. Lemma.- With the above notation 

A := XjUi'l - jiin'l > ^ai(cri -pi) - J^PzQ(m)(ai -pi) 

ill I 

where Ci{m) = Q(m2, . . . ,mj,). 



Proof. By replacing dimensions and inserting the estimate for — n'l as in we get 

ill I 

> Y'^ii^^i- Pi) -YPiCi{m)i(ri- pi). 

i I 



□ 



7.4.2. Corollary.- Let A = (Ai, . . . , A,., — pi, . . . , — Ps) be a polarization for W and let 



A = («!, . . . , — pi, • • • , — Ps) he the associated polarization for V as in \7.^ . If all 

tti > and 



A2 > 



Pici{m) 



then 



Cr^V-(Gz. X G«, A) = A) and Ci^\Gl x A) = W\G, A). 



Proof. The proof is the same as for |7.2.2| , because the spaces P/ and Pi{M") are defined 
in the same way and we thus get the estimate ((J2 — P2)o2i > en — Pi- n 



7.5. Estimate for C2 

The analogous estimate for (2 follows by duality while we can assume that s = 1 or r = 1. 
The proof could be done by formally transform it into a dual situation which is similar to 
that of [ni , but it is better to keep direct track of the weights. Let (x2, ... , x^, 71, . . . 7^) 
be given in © V and assume that its image under (2 is not semi-stable. Then there 
are subspaces P/ C Pi and Q'l C Qi such that 

x,{P: ® C PU , l{Pl ® Hli) C Q's , ® Bi+i^i) C Q'l, 
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where 7 is defined as in [773| , and such that 

i I 

We assume that all aj > 0, and then we may assume that P/ is maximal, i.e. the inverse 
image of P/_^ ® under Pj ^ Pj_i C?) v4*j_]^ for i > 2, and similarly P{ in Pi under 

Pi — ^ Qs ® -f^si- As in we can find subspaces N[ C A^"; such that 

Q[ = N[ © X; and hence {Qi/Q\Y = (Ni/N^Y © X^. 

We choose subspaces iV/' C X/ which are maximal such that 

Qi{N") c Q; C QliN'). 

We have X^' = We let Pf be the inverse image of Qs{N") under Pi ^ © Hsi- 
Then P{' C P[. Furthermore we let inductively P/' C P/ be the inverse images for i > 2. 
Then we have injections 

{p:/pn © A,-i - iii/i^i 

and induced by factorization the images 

i^'/i^" © A,,_i © . . . © A21 ^ (P;/i^") © Aa ^ Pi'/i^i". 
The induced injections 

imply the dimension estimates 

Pi -Pi < fliibi ~Pi) 
for ? > 2. Next we consider the homomorphism 

Zi= ® {Ni/N['y © if,* ^ © {Ni/N['y © P,; © Hl^. 

1<S 1<S 

We have Xg C ®{Ni/N[')* © P^^ and consider the subspace 

Fi=(5i^(X,©i/:i)cZi. 
By the definition of the constant di{n) = di{ni, . . . ,ns-i) we get 

dim(Zi/Yi) < di{n) codim(Xs) = di{n){as — Ps) 

where 

ai = codimiQ* (N/N'), Q*{N/N")) and pi = codim{Q* (N/N'), (Qi/Q'iY). 
Further we have a surjective map 

Z^/Y^ ^ (Pi/Pf)7(Pi/P0* 
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which is induced by the map Q*^H*^ — > Pj* and the induced surjection Q*{N/N")^H*-^ 
{P/Pi'Y, since A^^' = A^^. So we get 

p'l -p'i < di{n){as - Ps)- 
Now we can estimate the discriminant in 

7.5.1. Lemma; Let all the ai be non-negative and let A := 'Y^Uip'l — 'Y^pin'l. Then 

i I 

A > '^(3i{oi- pi) - '^aiaiidi{n){as - Ps)- 

I i 

Proof. Since ^ aipi = f^i'^^i ^-Iso have 



p'l) 



with the same steps as in the previous proofs we get 



Pi 



Inserting the assumption on the first difference and the estimate for p'^ — p'l we get the 
result. □ 



As in the previous cases we obtain the 

7.5.2. Corollary; In the above notation let all ai > 0, and all j3i > 0, and let 

i 

Then 

Q'W'\G,K)=Y'\Gl^Gr,K) and ^'^'{^,1) = Y\Gl ^ Gr,K). 



Proof. : In the notation of there is a surjection ((5s-i(A^')/'5s-i)* ® 
{Qs{N')/Q'g)* because A^" = A^^. Therefore {cTs^i — Ps-i)bs^s~i > (^s — Ps- If the condition 
of the Corollary is satisfied, then A > follows, where we use ps-i = Psbs,s~i + Ps-i- □ 



Combining the results of 7.4.2 and 7.5.2 we get the 
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7.5.3. Proposition; Let A = (Ai, . . . , Xr, —fii, . . . ,—fJ's) be a polarization for {W,G) 
and let A = (ai, ... , a^., —Pi, ■ ■ ■ , —Ps) be the associated polarization for (W, G). Suppose 
that all ai > 0, and all f3i > and that 

A2 > ^21 ^ fJ'iCiim) and fXs-i > bs,s-idi{n) ^ aiUn. 

I i 

Then 

C^W'iG, A) = W''{G, A) and C"^W^(G, A) = W'{G, A). 



8. Projectivity conditions 

The projectivity of tlie quotient in |6.1.1| depends on the second condition in (2), i.e. 
whether the boundary Z \ Z of the saturated set contains no semi-stable points of W. 
Again this condition depends on the chosen polarization and conditions for the weights. 
In order to derive these conditions in some cases we describe the boundary in terms 
independent of the group action. 

8.1. Saturated boundary. 

The elements of W are tuples w = {x2, ■ ■ ■ ,Xr,^,yi, ■ ■ ■ ,ys-i) of linear maps 

® ^ p,_i, Pi ® h:, ^ Qs, Qi+i ^ ® Qi 

If w G 2', there are an element w &W and automorphisms G Aut(Pj) , ai G Aut((5i) 
such that 

Xi = Pi^io^io{pT^ ®id), 7 = (Ti 07(1^) o (p^i yi = {id ® ai) o 7]i o a^]^^. 

Here id stands for the different identities of the spaces A, B and H. We let Xi respectively 
be the mapping 

Pi ® Ai^i^i ® ...®A2i^ Pi^i ® Ai_i,i_2 ® . . . ® A21 

induced by Xi respectively for i > 3. From the relations between the Xi and it follows 
easily that for each z > 3 the composition X2 o x^ o ■ ■ ■ o Xi has a factorization 

Pi ® ® ■ • ■ ® A21 ^ Pi 

:: ^-^"^i^ii^^^^ 
Pi ® Ail 

where the vertical map is the surjection induced by the pairings. This follows from the 
commutative diagrams induced by the automorphism pi and because ^2°^3°' ■ ■^(.i admits 
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such a factorization for each i > 3. We put X21 — X2- By the dual description for the 
maps yi we are given factorizations 




. . ® ® Qi 

of the maps yi o . . . o ys-2 ° J/s-i for Z < s — 2. By similar arguments there are also 
factorizations 



{Li) 



HI 



for all i > 2 and dually factorizations 




Qi ^ Ha 
B*i ® ® Hsi 



7 Bis^'d 

for all /. Moreover, there are further factorizations of the induced composed maps 




(Lii) 



and dually 




(Rii) 



7(1 (Slid 



All these factorizations are based on mappings induced by the pairings. All factorization 
conditions are independent of the chosen automorphisms. One can rediscover the original 
components (pu of w from or "ifn if Xj = and yi = rji for all j and all /. In fact we 
have 



8.1.1. Lemma; Let w — {x2, ... , x^, 7, yi, . . . , ys-i) G W. Then w & Z if and only if 

(1) rank Xi = J2i<j fori>2 
(1*) rank yi = Y.k<i bi+i,knk for I < s - 1 
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(2) X2 o o . . . o Xj has a factorization Pi ® An Pi for i > 3 

(2*) yi o . . . o ys_2 ° Us-i has a factorization Qg B*^ (8) Qi for I < s — 2 

(3) 7 o [xii ® id) has factorizations (Li) and {Lu) 
(3*) iyis <^ id) o J has factorizations (Ri) and (Ru). 

Proof. If w G the three conditions are satisfied by the above, where rank Xi is the 
dimension of the image of and rank yi is the rank of rii as the map Qi+i — > Bi_^_^ ^ (g) 
Qi. Let conversely w satisfy these conditions. We proceed by descending induction to 
find automorphisms pi by which the Xi can be identified with the ^j. Note that the 
factorization conditions are maintained under automorphisms. Since Xr has maximal 
rank it is an injection Mr ® ^r,r-i M^-i © Mr ® ^r,r-i = Pr-i- Hence we can find 
an automorphism pr-i of P^-i such that p^-i ° becomes S,r- Let us assume now that 
modulo some automorphisms Pr-i, • • • , Pi have Xj = for j > i. We are going to find 
an automorphism pj_i such that pi-i o Xi = Because of the rank condition we can 
assume that ®i<jMj ® Aj^i^i is the image of xi in Pi-i. Now using all the Xi o o . . . o 
we find that Xi has a factorization through the standard map 

Pi O ®Mj® Aj- i_i ^ Mi_i © © © Aj- 

induced by the pairings. Now the rank condition implies that Xi induces an automorphism 
on ®i<jMj © Aji_i. This can be used to make Xt the identity via an automorphism 
Pi_i. Now Xi = ^j. By the analogous dual procedure we can also find automorphism 
ai G Aut(Q;) such that we can assume that yi = rji. Finally the factorizations (Lu) 
or (Rii) resulting from (3) and (3*) yield mappings or \E';j from which we get 4>ii as 
composition 

m,®h;,>-^p,®Hu^Qi-^Ni. 

It follows from the special type of the and r]i that these are original components of an 
element w = {(pu) inducing 7(w) =7. □ 



8.1.2. Corollary.- With the same notation as in \8. 1 . i| , if w E Z \ Z , then 

(1) rank Xi < rank S,i and rank yi < rank rji with strict inequality for at least one i or I, 
and 



(2), (^), (3), (T) q/ [01 are satisfied. 

Proof. All conditions are closed and thus hold for points m Z. E Z \ Z then by |8.1.1| 
equality in (1) cannot hold for all i and I. □ 



We are going to derive effective sufficient conditions for the projectivity of the quotient 
in the cases (2, 1), (2, 2), (3, 1). 
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8.1.3. Proposition; Let the polarizations A and A be as in proposition \7. 5. Sj and let 
Z = G({W). Then Z \ Z contains no semi-stable point in the following cases 

(i) (r, s) = (2, 1) and A2 > Ci(m2)a2iAti 

(ii) (r, s) = (2, 2) and 



A2 > (/iiCi(m2)+/i2(c2(m2)-62iCi(m2))a2i, fii > {Xi{di{ni))-d2{ni)a2i)+X2d2{ni))h 



'21- 



Proof. We present only the case (ii), case(i) is an easier version of (ii). Let {x2, 7, yi) G Z\ 
Z and let us assume that rank X2 is not maximal. Let K be the kernel of M2 ® v42i Pi 
and let M2 C M2 be the smallest subspace such that K is contained in M2 ^ A21. We put 
P^ = M'2, 



P[ = X2{M'2 ® A21) , Q'2 = l{P[ ® H;^) and Q[ = ?/i(Q^ ® ^21) 



and consider 



A = aip[ + a2P>2 - Ml - M2- 



By definition p'^ = dmi{M2® A21 1 K) . Diagram (L2) reduces in our case, with M2 replaced 
by M'2, to 



M'. ® A21 ® h;, ^ P[ ® h;, - 



S2 




M'2 ® H^2 



and 722 vanishes on S2{K ® -^21) because K is the kernel of X2- Therefore 

q'2 < dim(M^ H*22/HK ® H;^)) < C2im'2)p[. 

In order to estimate q[ we consider diagram {L21) enlarged by the commutative square of 
induced pairings 



® A21 



ML(g)A 



h;^ ® ^21 



M'2 ® H^2 



M' ® H* 



B21 Q'2 ® B21 g; . 



'2 ^ ^21 -'-'11 ^ J"2 ^ -'-'12 

Again the map $12 vanishes on 6i{K ® H^^) and we get 

q'^ < dim(M^ ® Hy6,{K ® H*,,)) < ^(m'^)^;. 
Now we have the estimate 

A > 02^2 + («i - Aci(m2) - P2C2{m2))p'i. 
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Therefore the condition ai > /3iCi(m2) + /52C2(m2) would be sufficient, because a2P2 > 0. 
We modify the last estimate as follows. Since the weights in case (2, 2) are related by 

Ai = «! , /^2 = f32 

, and ^ „ , 

A2 = a2i«i + 02 fii = Pi + P2O21 



and since we have 



we get the estimate 



A2 — 021A1 > and ^2*^21 — p'l > 0, 



A2 

A > ( yUlCi(m2) - /i2C2(m2) + fi2Cl{m2)b2l)p'i. 

a2i 

This shows that A > if X2 is degenerate and the ffist condition of (ii) is satisfied. In 
case rank yi is not maximal the second condition follows by the dual procedure. □ 



8.2. The case (3,1) 

In order to derive a similar result in case (3, 1) we introduce the additional constant 03(7713) 
analogous to €3(1713) := ci(0, ms) in ^IA\ . Let 

M3 ® A32 ® H*,2 ^ M3 ® i/i*3 

be the linear map induced by the pairing and let /C be the family of all proper subspaces 
K C M3 ® A32 which are not contained in M3 ® ^432 for any subspace M3 C M3 different 
from M3. We put 

4(.,„3) ^ sup ^om<^H;^) 

KGJC codim(A) 

For brevity we write C3 = 03(7/13), C3 = 03(7713) = 01(0,7773) and Ci = 01(7772,7773). 



8.2.1. Proposition.- Let (r, s) = (3,1), let A = (Ai, A2, A3, — /xi) &e a polarization for 
{W,G) and A = (ai, 0:2, 0:3, — /ii) be the associated polarization for (W, G), and assume 
that all ai > 0. (In this case fii = :^.) If 

(1) a2C3 + A1C3 > yUiC3C3 

(2) A2 > a2i/iiCi 

(3) A3 > a3i;UiCi 

then Z \ Z contains no semi-stable point. 

Moreover, condition (1) may be replaced by any of the conditions 

(i) A3 > /iiC3a32 + a3iAi 

(ii) A3 > /iiC3a3i + 032^2 

(iii) A3 > /iiC3a32a2i 
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Remark: Z \ Z contains no semi-stable point also in each of the following cases 

(a) Ai > /iics 

(b) a2 > /iiCg 

(c) as > fiicsasi or as > /iiCgasa- 

This can be seen by a direct estimate of the discriminant A after substituting for q[ in 
the following proof. 

Proof. Let (0:2, a;s, 7) G Z \Z. We distinguish the following cases of degeneracy of X2 and 
X3. 

case 1: X3 is injective: Then by the proof of we can assume that X3 = ^s is the 
canonical embedding and that X13 and X2 have a factorization X2 in the following diagram 



Here also ^3 is the canonical embedding. Moreover it is easy to verify that in this case 
also the composed map 7 o (x2 (8> id) admits a decomposition 

(M2 ® A21 Hl^) ® (Ms ® Asi ® H*^) ^ (M2 ® iJ*2) © (M3 © H*^) ^ Qi. 

Here K = Ker{x2) 7^ since X2 cannot be injective by the assumption on its rank. We 
choose subspaces M2, M3 such that 

C M2 © A21 © M3 © A31 

and such that these subspaces are minimal with this property. Now we consider the spaces 

P!, = M'„ P^ = M^©(M^©As2), P[ = X2{P^®A2i), Q[=^{P[®H*,) 

and their discriminant 




M3 © A^r^ 




M2 © A21 © Ms © A 



Pi ■ 



31 



A = aip[ + a2P2 + «sP3 - PiQv 



By the definition of the constant Ci{m2,'m'^) and the diagram 

(M^ © A21 © M;^ © A31) © H*^ — >^ P[ © H* 




© © M;^ 



®H* 



we obtain the estimate 



q'l < Ci{m2,m'^)p[ < Ci{m2,m3)p[ 
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where by the definition of P[ we have p[ — m^a'z + 771.3031 — k. Inserting this we obtain 

A > {ixici - \i)k + (A2 - A*iCia2i)m2 + (A3 - //iCiOaOmg 

If fiiCi — Ai > 0, conditions (2) and (3) imply that A > 0. If, however, Ai > niCi we have 
the direct estimate 

A > (Ai - iiici)p\ + a2P2 + "3P3 > 0- 
This proves the proposition in the first case. 
case 2: xz is not injective 

Here we let K denote the kernel of xs and we choose a subspace M3 C M3 such that 
K C M3 (8) A32 and M3 is minimal with this property. Then we consider the subspaces 

P;^ = M^, P2' = 3;3(M>^32), Pi' = X2(P^®^2i), g; = 7(A'«'^ri)- 

We have the exact sequences 








K 
L 



M' (g) A32 



A 



X2 



Xl3 



31 



Pi 











where L denotes the kernel of Xi-^. From the factorization properties restricted to the 
spaces P/ and Q[ we extract the following commutative diagram of surjections 



(8) ^31 (g) H*^ 

Si 



7 



(g) A32 ® A21 H*i 



M'®H 



713 



M'^ (g) yl3 



13 



±32 -i 2 ^ -'-'12 

Prom this we get again the estimates 

?i < 03(777,3)^1 < 03(7773)^1 and ?i < 4(7773)^2 < 4(7773)^2, 
where p\ — m'^a^i — I and p'^ = 7/73032 — k. Let < i < 1 be a real number. Then we have 

q[<tc'^p'2 + {l-t)csp[. 
Substituting this into the discriminant we get 

A > (Ai - (1 - t)iiiC3)p[ + {a2 - tiiic'^)p2 + a^m'^. 
Now condition (1) enables us to find t with 



Q'l 



p' 
-'2 



712 

H* 



A 

A*lC3 



< t < 



0L2 

laic's ' 



such that the first two terms of the estimate are non-negative. Therefore A > 0, and 
again (2:2, X3, 7) is not semi-stable. 
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In order to show that (1) can be replaced by one of (i), (ii) or (iii) we substitute aj and 
p[ and get after cancelation 

A = —Xil — + Asmg — fiiq[ 

= -Xil + {fiic'^ - a2)k + {Xs - fiic'^as2)m'^ . 

If a2 > f^ic'^, then by a direct estimate we get A > 0. Therefore we may assume that 
/iiCg — 0:2 > 0. Since in addition / < m'^a^i, we get 

A > (A3 - fiic'^a32 - a3iXi)m'^. 

This shows that (1) can be replaced by (i). In the same way one shows that (1) can be 
replaced by (ii), using the other estimate of q[. That finally (1) can be replaced by (iii) 
can be shown by substituting first m'^ > ^ and canceling 02^2 and then substituting 

P2 > ^ to get 

a32a2iA > AiPi(a32a2i - 031) + (A3 - fiiC3a32a2i)p'i- 

□ 



8.3. Proof of theorems 1.5.1 and 1.5.2 



Theorem |1.5.1| is an immediate consequence of proposition |6.1.1| , corollary [7.2.2| and propo- 
sition |8.1.3|. Theorem |1.5.2| follows immediately from theorem |1.5.1| and OTT . 



9. Examples 

9.1. Constants 

We give here some constants (cf. used in the examples. The following result is proved 



m 



121 , prop. 6.1. 
9.1.1. Proposition .■ For homomorphisms of type 

(Ml ® C(-2)) © (M2 ® C(-l)) ^ A^i ® C 

on a projective space of dimension n we have 

mim — 1) n + 1 

ci(m) = — — -, ^ if m < n + 1, cAm) = — if m > n + 1, 

^ ^ 2(m(n+l)-l) - ' ^ ^ 2(n + 2) 
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9.1.2. Lemma; For homomorphisms of type 

Ml ® 0{-d) © C(-2) © ^ iVi © C 

on the projective space FV the constant ci(l, 1) is dim{V)/ dim{S'^~^V) . 



Proof. We put s{p) = dim(S'^\^). The homomorphisms 5i of [7.1| reduces here to the 
canonical map 

If is a proper subspace of S'^^'^V* © S'^^^V* which is not contained in one of the 
summands, it contains elements (/, g) with / 7^ or elements (/, g) with g 0. But since 
/ © S'^V S'^V is surjective, the map S{K) — > 5*^1^ is surjective. Hence codim{S{K)) < 
If K contains an element {0,g) with g ^ 0, then S{K) = S'^V © V. For then 
S{K) contains V, and since S{K) S^V is surjective, if follows that S{K) = S^V © V. 
Therefore, if codim{6{K)) > 0, there is a basis {fi,gi), ■ ■ ■ , {fkiQk) of K with /i, . . . fk 
linearly independent, i.e. dim(i^') < s{d — 2) or codim(i^') > s{d — l). Therefore ci(l, 1) < 
s{l)/ s{d — 1). But now we can find subspaces which realize this bound. For any z & V* 
we let K be the space of all (/, fz)J e S'^-'^V*. Then K = S'^-^V* and it follows also 
that in this case 6{K) = S^V. Then codim(5(ir))/ codim(is:) = s{l)/s{d-l). □ 



9.2. First example of type (2, 1) 

We use the abbreviation mJ-" for C" © JF for a sheaf and a positive integer and consider 
here homomorphisms 

20(-2)©C(-l) ^^^30 

over P2 of type (2,1). The polarization A = (Ai,A2,— /ii) is supposed to be proper 
for W and W, i.e. Aj > and > for all i. The only constant involved here is 
Ci(m2) = c(l) = 0. Therefore the conditions of |7.2.2| and p.l.3| are automatically satisfied 



by a2 = A2 — 3Ai > 0. Hence all the quotients of W'^^{G,A) will be good and projective 
under this condition. Since 2Ai + A2 = 1 and 3/ii = 1, we can replace the polarization by 
the rational number t = A2 > | (cf. \).'d\ ). The numerical condition for (semi-) stability 
then becomes 

1 1 . ^ 

A = -——mi +tm2- -n<0 (< 0), 

where {mi,m2,n) is the dimension vector of a (01, 02)^invariant sub-family of vector 
spaces, such that mi < 2, 777,2 ^1? 77 < 3. One can easily check that t = | is the only 
value for which A might be zero, and this is the case for the values (0, 1, 2) and (2, 0, 1). 
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And indeed, the homomorphisms (f) given by matrices 

**0\ / zi\ 

* * Z2 \ and ^2 

* * Zs J \ * * Zs J 

with generically chosen entries and hnear forms Zi are semi-stable and not stable for t = |. 
9.2.1. The case t > | 

It is easy to show that in this case (0i, ^2) is t-stable if and only if 

• 02 is nowhere zero 

• for any {ipi,ip2) = ^-(01)02) with h & H and any 1-dimensional subspace Mi C 

we have ^i(Mi(®0(-2)) ^ 0. 

The first condition says that coker(02) is isomorphic to the universal quotient bundle Q 
on P2, and the second that 0i induces a 2-dimcnsional subspace of H^Q{2). It follows 
that the sets W^{t) of stable points are the same for t > |, which we denote by W^. 
Moreover, from the above characterization of stable homomorphism we deduce that the 
geometric quotient M+ = W^/G is isomorphic to the Grassmannian 

M+ = Gr{2,H^Q{2)) 

which is smooth of dimension 26. There is an interesting subvariety Z C M_|_ which 
consists of the images of the homomorphisms 

/ \ 

p ;22 (1) 

which belong to W^. These are those (0i,02) for which the induced homomorphism 
2C(— 2) — > Q is not injective. We will see next that Z is isomorphic to the non-stable 
locus of Mo below and is smooth of dimension 10. 



9.2.2. The case t = | 

We write Wq"^ for When considering the matrix representations we find that 

C Wq^ and that the remaining part Wq^ \ consists of those homomorphisms for 
which 02 is zero in exactly one point. Such homomorphisms are equivalent to matrices 

(* * z \ 
* * w \ (2) 
f 9 J 

where z, w are independent linear and /, g are independent quadratic forms. Note, how- 
ever, that Wl intersects the non-stable locus of Wq^ in matrices equivalent to those of 
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type (1). But the orbit closures in Wq^ of both types (1) and (2) of matrices contain the 
direct sums 

z \ 

w \ (3) 
f 9 J 

of independent hnear and quadratic forms. Prom that it follows that the induced mor- 
phism 

M+ Mo 

of the quotients is bijcctive and moreover an isomorphism by Zariski's main theorem, be- 
cause both spaces are normal. The points of the non-stable locus MqxMq arc represented 
by matrices of type (3). It is again routine to deduce from this observation that 

Mo \ Mo" ^ P2 X Gr(2, H°0{2)). 

The subvariety Z C corresponds to this set under the isomorphism. We can also 
identify the set Mo" of stable points with Gr{2, H^Q{2)) \ Z. 



9.2.3. The case | < t < | 

Similarly to the case we find that here W!_ — W^{t) is independent of t and that 
Wf_ C Wq^. The remaining part consists now of all homomorphisms which are equivalent 
to a matrix of type (1). Note that now homomorphisms of type (2) are contained in Wi. 
The induced morphism 

M_ ^ Mo 

is again surjective but not injective over MoxMq. Let Y be the inverse image of MqxMq. 
Then Y consists of the points which are represented by matrices of type (2) which are 
not equivalent to matrices of type (3). It is easy to check that the restricted morphism 

M_ \ y — > Mo" 

is bijcctive and therefore also an isomorphism by Zariski's main theorem. We are going 
to verify that K is a divisor in M_. There is a morphism 

Y^P, 

which assigns to the class of (0i, ^2) the point x at which 02 is degenerate. In this case 

coker(02) ^ OeX,(l) 
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where Xj. is the ideal sheaf of x. For such ^2) we are given an exact diagram 



2C(-2) O 

. 20{-2) © 0{-l) ^""""^ 30 T 

Oi-1) SO O © J,(l) 

2C(-2) 



such that (01,02) corresponds to a 2-dimensional subspace T C H^{0{2) ©X,;(3)). The 
condition of defining a element of Y is that F is neither contained in if°Xj;(3) nor in 
H^{0{2))s for any section s of O © J^(l). We let C Gr{2, H'^{0{2) ® denote 
the open subvariety of such F. By assigning to F the class of (02, 02) where 0i is defined 
by a lifting in the above diagram, we get a morphism Ux M_ whose image is the fibre 
= p~^{x). The morphism 

Ux ^ Yx 

is nothing but the quotient of Ux by the algebraic group Aut((9 ©Xt;(l)). It follows that 
Yx is a variety of dimension 23. Using the techniques of this paper for this quotient, we 
can even prove that Y is smooth. Finally Y has dimension 25 and thus is a divisor in the 
irreducible and normal variety M_. 

Remarks: (1) One would like to interpret the matrices of type (2) as representing exten- 
sions of the sheaves coker(/, g) and 2x{l) = coker . Indeed a matrix of type (2) defines 
such an extension, but this extension is isomorphic to the direct sum. 

(2) The above correspondence between (0i,02) and F indicates that the quotient spaces 
considered here are spaces of coherent systems as in |^ . 

9.2.4. The flip 

The diagram M_ — > Mq <^ M+ can be interpreted as a flip. It is induced by the inclusions 
Wl C Wq" D W^. The orbits of stable points of type (2) in Wl and of type (1) in 
don't intersect in Wq^ but so do their closures in Wq'^. Thus the fibres of M_ —>■ Mq and 
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Mq <— M_|_ correspond to the two different types of semi-stable orbits in W^^ defining the 
same points in Mq \ MJ. 

9.3. General homomorphisms of type (2, 1) 

In a more general situation of type (2, 1) we consider homomorphisms 

miC(-2) © m2C(-l) n^O 

over P„. A polarization in this case is determined by the rational number t = 1712X2 with 
< t < 1 and 1 — t = miAi, fii = l/rii. A A-(semi-)stable homomorphism is then called 
t-(semi-)stable. We write W{t) and W'{t) for W''{G,A) and W'{G,A). In terms of t 
the conditions are 

(n + l)m2 , (n + l)m2 

1 > t > , ' — and t > ^ ^ ci m2 ■ 

[n + l)m2 + rrii rii 

The constant Ci(m2) is given in proposition |9.1.1| . Such polarizations exist if and only if 

ni > (n + l)m2Ci(m2). 

In order to measure t-stability we introduce the numbers 

ri = — , r2 = — , S2 = — 
nil ^2 ^1 

and call (ri,r2,Si) 0-admissible if there are subspaces M[ C Mi,M2 C M2,N[ C A^i of 

dimensions m[, mj, n[ such that maps M[ (g) 0(-2) © M2 ® C(-l) into N[ © O. Then 

is t-(semi-)stable if and only if for any 0-admissible proper triple (ri, r2, si), i.e. a triple 

which is neither (0, 0, 0) or (1, 1, 1), we have 

At = (1 - t)ri + trs - si <0 (< 0). 

A polarization t is called critical if there are proper triples with = 0. Thus the critical 
values of t are the rational numbers 

si - ri 
r2 - ri 

where we may assume si 7^ 0, 1 and thus ri 7^ r2. We let tmax be the maximal critical 
value if there are such with < t < 1 and put t^ax = otherwise. If t is not critical we 
have W'{t) = W''{t). 

9.3.1. Lemma.- Suppose that m2 and rii are relatively prime and that t^ax < t < 1. 
Then = (0i, ^2) is t-stable if and only if 

(1) 02 is stable with respect to the group GL(M2) x GL(A^i). 

j 

(2) For any 1 -dimensional subspace C ^ Mi, and any h G Hom(Mi © 2),M2 © 
0{—l)) the map {(fi + ho ip2) o j : 0{—2) ^ Ni ® O is not zero. 
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Proof. By the characterization of stabihty in section 3 the homomorphism 02 is stable if 
and only if for any proper pair M2 C M2,N[ C Ni of 02^admissible subspaces r2 < si. 
Now let (01,02) be stable. If 02 were not stable there would be a proper 02-admissible 
pair (r2,si) with si < r2. But then si < r2 because 1712, ni are supposed to be relatively 
prime. Then Si/r2 < t because Si/r2 is a critical value and thus At = r2t — si > 0, 
contradicting the stability of (0i,02). The condition (2) is trivially satisfied if (0i,02) is 
t-stable, because otherwise (1, 0, 0) would be admissible with At = 1 — t > 0. We have to 
show now that conversely (1), (2) imply that (0i,02) is t-stable. For this let (ri,r2,Si) 
be a proper (0i, 02)-admissible triple. If ri < r2 and r2 = 0, there is nothing to prove. 
If r2 > then r2 < Si by (1) and we have t{r2 — ri) < Si — ri and hence At < 0. If 
however r2 < ri we have Af < in case ri < Si. Since the case Si < r2 is only possible if 
■Si = ^2 = and then ri = by (2), we can assume that r2 < Si < ri. But then 

ri - r2 

because the fraction is a critical value, and last inequality is the inequality At < 0. □ 



Now we are able to describe the space M+ = VV^ (t) / G for t^ax < t which is independent 
of t. According to the lemma W^lt) can only be non-empty if there are stable morphisms 
02. This is the case if and only if 

1 ni 
— — < — < a(n) 



a[n) 1712 

where a{n) = i(n+l + A/(rH-Tp"^^), see p. We restrict ourselves now to the case where 
in addition to the previous conditions on ni,m2 we have ni > nm2 and (ni,m2) = 1. 
Then a stable 02 is injective and a subbundle (except at finite number of points in case 
rii = nm2, see [Q, [Q). The quotient space of this space of stable homomorphisms by 
GL(M2) X GL(A''i) is denoted by = N{n + 1, m2, ni). It is a smooth projective variety 
and there is a universal sheaf £^ on x P„. For x G A^ let Sx denote the fibre sheaf 
representing x. Since it is the cokernel of the representing homomorphism 02, we get 



;,»f.(2) = („ + i)(^ii!i±i)-™.) 



Therefore p^S{2) is locally free on A^ where p denotes the first projection of A^ x P„. Now 
M+ can be non-empty only if 

, , /ni(n + 2) 
mi < (n + 1) I m2 



If conversely this is the case for any stable 02 and any subspace Mi C H^Sx{2) where 
X = [02], there is a lifting 0i : Mi O 0(-2) ^ A^i O O of Mi ® 0(-2) 8^, and (0i, 02) 
satisfies (1), (2) of the lemma. It follows now easily by considering corresponding families 
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that 

where Gr^ denotes the relative Grassmannian. It is more difficuh to characterize the 
other moduh spaces M{t) = W'^{t)/G for the intervals between the critical values or for 
the critical values and to interpret the flips between them. 



9.4. Example of type (2,2) 

We consider now a simple example of type (2, 2) on P3 of homomorphisms 

C(-2)©C(-1) ^C©3C(1). 

Again the polarizations A = (Ai, A2, — /ii, — /i2) are supposed to be proper for W and W 
such that we have Aj > 0, /xz > and 

A2 > 4Ai and /ii > 4/i2- 

All constants Ci{m2) and di{ni) are again zero, because m2 = ni = 1. Then by the above 
conditions also the conditions for proposition [7.5. 3| and proposition |8.1.3| are satisfied, 
such that there exists a good and projective quotient W^'^i^G, A)/ /G for any polarization 
satisfying the conditions. Since we have Ai + A2 = 1 and fii + 3/i2 = 1, the polarization 
A is determined already by A2 and /ii, for which the above conditions become 

4 3 

1 > A2 > - and - > 1 - /ii > 0. (1) 

5 7 

Next we derive the conditions for the occurrence of true semi-stable points. If 
(mi, 7712, ni, 712) is the dimension vector of a ^-invariant sub- family we have to consider 
the equation 

A = (1 - A2)mi + A2m2 - /iini - -^(1 - fii)n2 = 0. 

By inserting all possible dimension vectors we get the 6 conditions 

3 3 3 

l-/ii = ^A2, l-;Ui = --A2 + - (2) 

for k = 1,2, 3. If one of these is satisfied, there might be non-stable points in W^'^IG, A). 
In the following figure 1 the lines with the equations (2) are shown together with the 
rectangle (1) (lower right), for the points of which we get good and projective quotients. 
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Figure 1 



The homomorphism 



defined by the matrix 



\ 



J 



where the Zi are homogeneous coordinates of P3, is easily verified to be G-stable for each 
polarization A in the rectangle (1). Therefore the moduli spaces are not empty. On 
each of the 3 lines in the rectangle (1) each point defines one and the same open set 
A) and hence one and the same moduli space with semi-stable and non-stable 
points. Similarly, on each of the 4 open triangles we have one and the same moduli space, 
which is a smooth projective geometric quotient. Each of the 7 spaces has dimension 77. 
The reader may also verify that the moduli space for an open triangle admits a morphism 
to the moduh space of each of its edges, thereby defining a chain of flips. 



64 



J.M.DREZET AND G.TRAUTMANN 



9.5. More general homomorphisms of type (2,2) 

More general homomorphisms for which we know the constants exphcitly are homomor- 
phisms of type 

miC(-2) © 20(-l) ^ 2C © ^20(1) 
over P3, say. By proposition the constants are here 



ci(2) = rf2(2) = ^ and 02(2) = d,{2) = ^. 

Let W be the space of those homomorphisms. A proper polarization 
A = (Ai, A2, — /ii, — /U2) for W satisfies 

miAi + 2A2 = 1 , 2/ii + n2/i2 = 1 

with Ai, A2, /ii, /U2 positive. We will also assume that 0:2 > 0, Pi > 0, i.e. A2 > 4Ai and 
/ii > 4/X2. These four conditions can be replaced by 

4 1 4 1 

< A2 < - and < /ii < - (1) 



8 + mi 2 8 + n2 2 

9.5.1. Claim.- There are polarizations A such that W'^'^{G,A) admits a good and projec- 
tive quotient in the following cases 

(i) mi < 6 and n2 < 8 
(i') mi < 6 and ^2 = 8 

(ii) 8 < mi + 3 < 77-2 and 8mi + 8 < 7n2 



Proof. The conditions of |7.2.2| for the equivalence of (semi-) stability become 



4 16 
A2 > ^{fii + 4/i2) and fii > y (4A2 - 15Ai) (2) 



and the conditions of |8.1.3| for the projectivity of the quotient become 



4 4 
A2 > - and /ii > -A2 . (3) 

The first condition of (3) follows already from the first of (2). After replacing Ai and fi2 
conditions (2) and (3) are equivalent to 

|^2A2 > (n2-8)/ii+4 

^mi/ii > {Ami + 30)A2 - 15 (4) 

/il > fA2 

Using (1) for A2, we find that (4) has a solution (A2,/ii) if the system 

8T^ > (n2-8)/ii + 4 

/^l > 7(8+mi) 
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has a solution jj^i. For this we distinguish the cases n2 < 8, n2 = 8, 8 < n2. If n2 < 8 the 
first inequahty of (5) has a solution /ii < ^ if mi < 6. If n2 = 8, then mi < 6, which is 
case (i'). If n2 > 8, the first inequality of (5) reduces to 

7n2-4mi-32 4 

> /ii > — . (6) 



(n2-8)(mi + 8) " n2 + 

Then (5) has a solution fii if and only if 

7n2 - 4mi - 32 > 
(7n2 - 4mi - 32) (ns + 8) > 4(^2 - 8) (mi + 8) 
7(7n2 - 4mi - 32) > 16(^2 - 8) 

These inequalities reduce to 

7n2 > 4mi + 32 
7n2 > 8mi + 8 
33^2 > 28mi + 96 

They are all satisfied if we suppose (ii) of the claim. □ 



In figure 2 the lines of the critical values of the polarizations i.e. of the pairs (A2, ^i) are 
shown together with the small region of those pairs which satisfy the sufficient conditions 
(4) for the existence of a good and projective quotient, based on the values mi = 3 and 
n2 = 5. 



9.6. Example of type (3, 1) 

As an example of type (3, 1) we consider only the space of homomorphisms 

C(-4)©0(-2)©C(-l) 



over P3. We assume again that all Aj and all ai are positive. Then the conditions of |7.2.2 
together with the normalization of the polarization are 

Ai + A2 + A3 = 1 /^i = i 

A2>10Ai A2>|ci(l,l) 
A3 - 4A2 + 2OA1 > 0. 

As additional condition for the projectivity of the quotient we use condition (a) of the 
remark following proposition |8.2.1] . Since in this case both the constants 03(1) and c'^{l) 
are zero, this condition is just Ai > and is already satisfied by our assumption. 

For homomorphisms of the above type the condition A3 < | is necessary if W^IG, A) 7^ 0. 
For if = {4>i,4>2,4'3) is an element of W then 03 has degree 1 and thus contains at 
most 4 independent components. Then mi = m2 = and rris = 1, ni = 4 is a choice of 
dimensions of 0-invariant subspaces and the discriminant becomes A = A3 — |. 
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By |9.1.2| the value of Ci(l, 1) is |. Now it is easy to see that there exist polarizations A 
which satisfy the above inequalities. That W^lGjA) is then indeed non-empty follows 
from the existence of generic matrices as in |9.4| . Moreover there are again regions of 
polarizations for which the sets ^^'^'^(G', A) are the same and which are responsible for 
flips. 




Figure 2 

Here the horizontal axis represents 1712X2 and the vertical axis represents ni/Ui 

for nil = 3 and n2 = 5. 
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10. Construction of fine moduli spaces of torsion free sheaves 



Let n,k be integers such that n > 2 and 

(n + l)(n + 2) 



< k < {n + lf 



2 

Let y be a vector space of dimension n + 1, P„ = ¥{V). We will study in this chapter 
morphisms of sheaves on P„ of type 

$ = ($1, $2) : 0{-2) ® ^ C(-l) © (O (g) C'^). 

Let 

/i : — > V* 

the linear map induced by $1. For semistable morphisms (with respect to a given polar- 
ization) /i is non zero. So it is of rank 1 or 2. Morphisms $ such that /i is of rank 2 are 
called generic, and those such that /i is of rank 1 are called special. 



10.1. Generic morphisms 

Suppose that $ = ($1,^2) is a generic morphism. Let P = Im(/i) and Pn-2 C P„ be 
the linear subspace of zeroes of linear forms in P. Then $1 is isomorphic to the canonical 
morphism 

0{-2) ^ P ^ 0{-l) 

hence wc have ker($i) ~ 0(— 3), and Im($i) ~ 2p^_2(— 1) (the ideal sheaf of Pn-2 
twisted by C>(-1)). Let 

: 0{-3) — >0(^C'' 

be the restriction of $2 to ker($i). It vanishes on P„_2 and induces a hnear map 

/':C'=*^i7°(Jp„_,(3)). 

10.1.1. Lemma ; //$ is semi-stable (for some polarization) then f is injective. 

Proof. Let Ko = ker(/')^ C C^. Then Im($') C O Kq. The morphism 

0{-2) (8) ^ 0{-l) © (O CV^o) 
induced by $ vanishes on 0{—3) = ker(<I>i). Hence it induces a morphism 

(^1,^2) :Xp„_,(-l) -^0(-l)©(0®CVi^o) 

where ipi is the inclusion. Since }iom{Ip^^_2{—l),0) = B.om{0{—l),0), we can (by 
replacing $ by an element of its Hom(C(— ® C'^)-orbit) suppose that ip2 = 0. It 
follows that Im($) C 0{-l) © (C ® Kq), and since $ is semi-stable, we have Kq = C'', 
i.e. /' is injective. □ 
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Note that we have taken k < {n + l^ — /i°(Tp^_2(3)), to allow the injectivity of /'. 

Suppose that /' is injective. Let K — Im(/'). then is isomorphic to the canonical 
morphism 

(l)K : 0(-3) — >0®K*. 

It is easy to see that P and K depend only on the G-orbit of $. Conversely, suppose 
P and K are given. We can define an element ($i,$2) of W associated to P and K 
as follows : let {zi,Z2) be a basis of P. Let {ziqii + Z2q2i)i<i<k be a basis of K, with 
92i £ S'^V*. Using this basis we can identify K and K* with C'^. We define 

$1 : 0(-2) — . 

by 

^2 ^y* 

(A, n) I \zi - fiZ2 

and 

$2 : 0{-2) ® — > O ® X* ~ O C'^ 

over a; G P„ by 

^2x{x^ ® {Klj)) = {M2i{x) + A*gH(a;))i<j<fe. 

10.1.2. Lemma ; Let K C H^{I^^_^{2>)) a linear subspace of dimension k. Then is 
injective outside of a closed subvariety of codimension 2. 

Proof. Let x G Pn- Then $x is non injective at x if and only if all the elements of K 
(which are homogeneous polynomials of degree 3) vanish at x. Suppose that is non 
injective on an irreducible hypersurface S. Then all the polynomials in K vanish on S. 
Let / be an irreducible equation of S. Then all the elements of K are multiple of /. It 
follows that / is of positive degree d < 3, and K C S^~'^V*.f. But this in impossible 
since 

T fT^\ (n+l)(n + 2) fr,2tr*\ 

dim(i^) > '-^ = dim{S^V*). 

□ 

10.1.3. Lemma ; Let $ = ($i,$2) be defined by P C V* and K C H^{1^^_^{?>)). 
Suppose that there exists a polarization such that $ is semi-stable. Then $ is generically 
injective anc? coker($) has no torsion. Moreover, if K is generic, $ is injective. 
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Proof. Lemma [10.1.2| implies that $ is injective outside a closed subvariety of codimension 
> 2. It follows that $ is generically injective and that coker($) has no torsion. To 
prove that $ is injective for a generic K, it suffices to find a K such that $ is injective. 
Let {zi, Z2) be a basis of P. Let qi,--- ,qr, (resp. q'l,-'' ^q's) be linearly independant 
elements of S'^V* that have no common zeroes in P,„, with r + s = k (this is possible since 
2n + 2<k< (n+ 1)2). Let 

K = (©^ig.)©(©^2g;). 

It is easy to see that for such a i^T, $ is injective. □ 



10.2. The obvious moduli space of morphisms and its universal sheaf 

Let P G V* a plane, Pri-2 C P„ the subspace defined by P and K C if°(Xp^_2(3)) a linear 
subspace of dimension k. Let S{P,K) = coker($), where $ is a morphism associated to 
P and K. Since the G-orbit of $ is determined by P and K, S{P, K) is well defined. We 
will give another construction of £{P, K). 



Let J^K = coker($^). It is a torsion free sheaf according to lemma |10.1.2 . 



10.2.1. Lemma .■ We have Ext^(Op„_2(-l), J^i<-) ~ C, and the non-trivial extension of 
Tk by C'p„_2(~l) is isomorphic to £{P,K). 

Proof. The exact sequence 

(*) — > C(-3) — >0®K* — >Tk — ^ 
implies i/°(.Fi^(l)) ^ V* ^ K% H^{Tk{1)) = {0}. Using the exact sequence 

Xp„_, 

we obtain the exact sequence 

0^Hom(O(-l),J-;^) = -^Hom(Jp„_,(-l),J-^) -^Exti(Op_,(-l),^;,) ^ 

From (*) we get H\J^k{2)) - S^V* ® K*, H^{J^k{5)) ~ S^V* ® K*/CiK, where zk is 
the inclusion K G S^V*. From the exact sequence 

Oi-2) 0{-l) ® P^* Jp„_, 

we deduce the exect sequence 

. Hom(Jp_,(-l),^A-) S^V* 0P^0K* S^V* K*/Cik 

where 6 comes from the multiplication 
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The kernel of /i is canonically isomorphic to A^P"*" ® V* and it is easy to see that ix is 
contained in the image of /i ® Ir* ■ It follows that we have an exact sequence 

— >V*(^K* — > ker(^) — > Cix — ^ 
and that Ext\Op„^^{-l), Tk) ^ C. 

The last assertion follows from the commutative diagram with exact rows and columns : 



C(-3) 0®K* J^K 



C(-2) ® C2 © (O ® K*) £{P, K) 



(^p„-.(-i) — -0 



□ 



Let M be the projective variety of pairs {P,K), where P is a plane of V* and K C 
if°(Xp^_2(3)) is a vector subspace of dimension k (P„_2 beeing the codimension 2 linear 
subspace of P„ defined by P) . We can view M as a moduli space for generic morphisms. 
We will give a construction of a universal sheaf K on M x P„, i.e E is flat on M and for 
every (P, K) G M, E(pj^) is isomorphic to the cokernel of a generic morphism associated 
to (P, K). It is also possible to define a universal morphism whose cokernel is isomorphic 
to E, but we will see this more generally in p.0.4| . 



Let Gr{2, V*) be the grassmannian of planes in V* and g : M ^ Gr{2, V*) be the obvious 
projection. Let U be the universal subsheaf of O x V* on Gr{2, V*). Let 

PM-.Mx P„ ^ M, p2 : M X P„ ^ P„ 

be the projections. Then we have a canonical obvious morphism of vector bundles on 
M X P„ 

Let V be its cokernel. It is a fiat family of sheaves on P„. For every (P, K) G M we have 
'P(p,K) = ^¥n~2- L^t /C be the universal sheaf on M x P„, such that }C(p,k) = K. Then 
we have a canonical obvious morphism of vector bundles on M x P„ 
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Let T be its cokernel. Then for every (P, G M, J^(p^k) is the sheaf that was noted J^k 
before. By lemma [1021 , the sheaf Sxtl^iV ® J^) is a hne bundle L on M 

Then we have a universal extension 







E 







on M X P„. Then using lemma 10.2.1 it is easy to see that for every (P, K) G M, E(p^x) 
is isomorphic to the cokernel of a generic morphism associated to (P, K). 



10.3. Special morphisms 

Let $ = ($1, $2) be a special morphism. Let /i : ^ V* the associated application of 
rank 1. Let H be the hyperplane of defined by Im(/i). We have an exact sequence 

0(-2) 0{-2) ® C2 ^ 0(-l) Oni-l) 0. 

Let 

— > H%OHi2))(^C'' 

be the linear map induced by $2- 

10.3.1. Lemma .■ //$ is semi-stable (for a given polarization) then $2 is injective. 

Proof. Let Ci, C2 be the two factors C of C © C = C^. We can suppose thet ker(/i) = Ci. 
Let 

$2i : C(-2) ® Q — yO^C'' 

i = 1,2, be the restrictions of $2, defined by qu,--- ,qki G S'^V*. Let (zi,--- , Zn+i) 
be a basis of V*, such that zi is an equation of H. By using the action of 
Hom((9(— 1) (8) C^, O © C^) on W we can assume that ^21, ■ " " , Q'2fc G 5*^ < 2:2, ■ ■ ■ , Zn+i >• 

Now $2 is not zero on Ci : otherwise we would have Qu G ZiV*, and Im($2) C O ® C^', 
with 

k' < n + l + dim{S^ < ^2, ■ ■ ■ , z^+i >) < dim(5V*) < k, 

and this would contradict the semi-stability of $. Hence, by considering the action of 
GL(2), it suffices to prove that $2 does not vanish on C2. Suppose it does. Then $2 
vanishes on 2)®C2, because q2i E S'^ < Z2, ■ ■ ■ , Zn+i >, and again Im($2) C O ® C^', 
with k' < dim(S'^V"*) < fc, which contradicts the semi- stability of $. □ 



10.3.2. Lemma .■ Suppose that $ is semi-stable with respect to some polarization. Then 
it is injective outside of a closed subvariety of codimension > 2, and coker($) has no 
torsion. 
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Proof. It suffices to prove the ffist statement. Let x G P„ and m G such that 
(g) u) = 0. Then we have either m G Ci or m ^ Ci and x E H. Suppose that $ 
is not injective at all points of an irreducible hypersurface D ^ H. Then the same is 
true for $|c'(-2)®Ci- Suppose that this morphism is defined by quadratic forms gi, ■ ■ ■ , g^. 
These forms vanish on hence they are all multiple of an equation oi D. It follows 
as in the proof of |10.3.1| that Im($2) C O^C^', with k' < k, which contradicts the 
semi-stability of $. 

Now it remains to prove that $2 is generically injective on H, but this follows easily from 
the fact that $2 is defined by an injection H^{Oh{2)) ® C*^. □ 



10.4. Fine moduli spaces oe torsion-eree sheaves 

10.4.1. Definition : Let 5 be a smooth variety, JF a coherent sheaf on S" x P„, flat on 
S. We say that S* is a fine moduli space of sheaves with universal sheaf if the following 
properties are verified : 

(i) For every closed point s G 5* the Kodaira-Spencer map 

Us : TsS — > Ext^{J^s,^s) 

is bijective. 

(ii) For every closed points Si, S2 G S" with Si 7^ S2, J-'si and jF^j are not isomorphic. 

(iii) For every flat family S of coherent sheaves on P„ parametrized by an algebraic variety 
T, and for any closed points s E S, t E T such that J^s — ^t, there exists an open 
neighbourhood U of t in T, and a morphism f : U ^ S such that f{t) = s and 

(/ X /pJ*(.F) ^ £p. 

(cf. 0). 

For example moduli spaces of stables sheaves admitting a universal sheaf are fine moduli 
spaces of sheaves. 



10.4.2. Application of theorem \1.5.i 
Polarizations for morphisms 

C(-2) ® — ^ ®{0® 

are defined by pairs (Ai, A2) of positive rational numbers such that A2 + AiA; = 1 (so here 
A2 is associated to C(— 1) and Ai to O ® C*^). 

By theorem |1.5.2| , there exists a projective good quotient of the open subset W^'^ of 
semi-stable points as soon as 

^ - n+l + k- 
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The critical polarizations in our range are given by 



Ai — — , t — X2 
2p 



2p' 



n + l + k (n + l)(n + 2) 
<P < 



Let 



(n + l)(n + 2) 



n+l + k 



+ 1 



(where [x] denotes the integer part of x). Then we obtain exactly q moduli spaces of 
morphisms corresponding to non critical values : Mi, ■ ■ ■ , Wig, where for 1 < i < q 



M{t) for t = l 



1 

2^ 



with p = i + ^ e beeing a sufficiently small positive rational number. We have 

Mg = M (cf. the end of llO). 



10.4.3. Fine moduli spaces 

Suppose that we choose a polarization such that t is not a critical value. In this case we 
have W^** = W^, and the stabilizer in G of the points of is the canonical subgroup 
isomorphic to C Let M{t) = W^/G, and vr : M{t) be the quotient map. On 

X P„ we have a universal morphism 

(where p2 is the projection x P„ — P„) such that = coker(\E') is a fiat family of 



torsion free sheaves on P„ parametrized by W'"^ (this is a consequence of lemmas |10.1.3 



and |10.3.2| ). There is a canonical action of G on such that C acts by multiplication. 

Recall that a G-sheaf S on x P„ descends to M{t) x P„ if there exists a coherent sheaf 
£' on M{t) X P„ and a G-isomorphism (vr x Ip^)*{S') ~ £. 



10.4.4. Theorem : There exists a G-line bundle C on M(t) x P„ such that T ® C 
descends to M(t). Let S be the corresponding sheaf on M{t) x P„. Then M{t) is a fine 
moduli space of sheaves on P„ with universal sheaf S. 

Proof. On we have a canonical action of G on the bundles Ow ® C^, L = Ow and 
Ow ® C^. On these bundles C acts as ordinary multiplication by scalars. Let ^0, be 
the G-bundles 

^0 = ip;iO{-2))®C')®p*^{C-'), Bo = (p;(0(-l))©(O®C^))®p^(£-i) 

(where pw is the projection W'"^ x P,„ —>■ W^). On these bundles C acts trivially. We can 
multiply the universal morphism with p^{L~^) and we obtain a new universal morphism 

*o : A — > Bo. 
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Now it is easy to see that the bundles descend to M{t) x P„ either directly 

from our construction of the quotient, or by using the more general results of fl^, 2.3. 
Let A = Aq/G, B = Bq/G. The G-morphism \l/o also descends and we get a universal 
morphism of vector bundles on M{t) x P„ 

^ : ^ — >B. 

We define now S = coker(\&), and it is clear that 7r*(£^) ~ JF L~^. 

Now we prove that the Kodaira-Spencer map ofSaiz^ M(t) is bijective. Let w G ^^^{z). 
Then we have a commutative diagram 

T^W T,M(t) 

Exti(J-^,^^) — Ext^(£,,£,) 

The tangent map Ttt is surjective because M(t) is a geometric quotient. So it suffices to 
prove that uj^ is surjective and that dim(Ext"'^(£^2, i^^)) = dim(M(t)). Consider the exact 
sequence 

— ^ = C(-2) ® — . i3o» = C(-l) © (C ® C'^) — ^T^ — > 0. 

It is well known that (up to a sign) uj^ is the composition 

Hom(A^,i3ot„) — > Hom(^o«>,^«>) — ^ Ext^(JF^„, JF^„) 

of maps induced by the preceeding exact sequence. Now the result follows easily from the 
exact sequence 

— ^ End(J-^) End(i3o^) ^ Hom(A^, i3o^)/ End(^o^) Y.-x\}{T^,T^) — ^ 0. 

We must now verify that if Z\^Z2 G M(t) are distinct closed points, then and are 
not isomorphic. This follows from the more general following result : if two injective 
morphisms of vector bundles on P„ 

C(-2) ® — > (C(-l) ® C"2) © (C ® C"!) 

have isomorphic cokernels, then they are in the same orbit. 

The property (iii) of the definition of a fine moduli space is easily verified. □ 



It follows that the q moduli spaces of morphisms Mi, ■ ■ ■ ,Mq, with their corresponding 
universal sheaves, are also fine moduli spaces of torsion free sheaves on P„. The moduli 
space Mq is the same as the obvious one M (cf. |10.2|) , and the corresponding universal 
sheaf is the same (up to an element of Pic(M)) as E. 



These examples are generalizations of the case of P2 (with k = 7) that was treated in |Tl 



But in this case our results are not needed, because we get only two fine moduli spaces : 



QUOTIENTS BY NON-REDUCTIVE GROUPS 
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one is the obvious moduli space and the other is the corresponding moduh space of stable 
sheaves on P2. 

On P„, n > 3, our moduli spaces are new. We don't know if the corresponding moduli 
space of stable sheaves is among them. 

10.4.5. Remcirk: it is not hard to prove that all the moduli spaces Mi, • • • ,Mq are 
distinct. 
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